System and method for implementing computer-simulated reality interactions between users and publications

ABSTRACT

Methods, systems, and media for enhancing one or more publications by receiving live video captured by a user, the live video comprising video of a publication, the publication comprising copyrighted content; identifying at least one first trigger in the live video, identifying one or more first three-dimensional, interactive media associated with the at least one first trigger and pertaining to the copyrighted content, and presenting to the user the first three-dimensional, interactive media; and identifying at least one second trigger in the first three-dimensional, interactive media, identifying one or more second three-dimensional, interactive media associated with the at least one second trigger and pertaining to the copyrighted content, and presenting to the user the second three-dimensional, interactive media to progressively deepen and enrich the engagement with the copyrighted content of the publication.

PRIORITY CLAIM

This application is a continuation of U.S. Utility application Ser. No.17/230,922 filed on Apr. 14, 2021 which is a continuation of U.S.Utility application Ser. No. 16/920,250 filed on Jul. 20, 2020 which isa continuation of U.S. Utility application Ser. No. 15/445,806 filed onFeb. 28, 2017 which claims the benefit of U.S. Provisional ApplicationSer. No. 62/345,320 filed Jun. 3, 2016, the entire contents of theseapplications is hereby incorporated by reference.

COPYRIGHT STATEMENT

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND OF THE INVENTION

Augmented reality (“AR”) technology provides a live direct or indirectview of a physical, real-world environment whose elements are augmented(or supplemented) by computer-generated sensory input such as graphics,video, sound, or GPS data.

Virtual reality (“VR”) technology, on the other hand, utilizes softwareto generate realistic images, sounds and other sensations that replicatea real environment (or create an imaginary setting), and simulate auser's physical presence in this environment, by enabling the user tointeract with this space and any objects depicted therein usingspecialized display screens or projectors and other devices.

SUMMARY OF THE INVENTION

Traditional physical publications do not enable the user or reader toexperience character or story interaction (other than imagination of theuser or reader). Traditional physical publications also do not enablethe user or reader to experience character or story interaction from afirst person point of view (e.g., from the perspective of a protagonistor an antagonistic), except in the imagination of the user or reader.Thus, traditional physical publications do not enhance the user's orreader's sense of cognitive and effective or affected empathy for thecharacters or the story in the publication.

Although AR technology has been used in the context of book readingexperience, such conventional AR books do not utilize AR as a bridgeinto VR book interaction experiences, nor do these conventional AR booksdynamically shift between AR and VR to provide the user or reader withmore immersive book interaction experiences. Hence, there is a need formore robust and scalable solutions for implementing computer-simulatedreality interactions, and, in particular embodiments, to methods,systems, and computer software for implementing computer-simulatedreality interactions between users and physical publications toprogressively deepen and enrich the engagement with the copyrightedcontent of the publication.

The technologies described herein including “Quantum Storey” books, aresignificant improvements over the e-book which is an evolution of theprinted book. Quantum Storey books allow the user to virtually stepinside a virtual simulation of the book, swapping their perception,subjectively, and sociality, overlaying a perceived experience takenfrom the written text and 2D imagery from the book and applying it intoan 360° 2D and 3D animated virtual world. This allows the user toexperience new perspectives through total immersion and enables theability to realize first person, an experience both good and bad to aidthe building of empathy. In addition, the technologies described hereinenable the user to step into the shoes of the other characters in theexperience to recognize how it feels to be in the situation from otherperspectives. These direct and visceral experiences and the process ofacquiring and storing these unique and applicative learnings enable amuch deeper perception, sensation, notion, and intuition of thesituation resulting in a stronger emotional and empathetic response tothe other person's situation.

For instance, a child reading a traditional book about a young personwho is being bullied or on the receiving end of the bullying behavior,can only read about how that person might feel through the words and tryto understand it visually through a motionless illustration. Thetechnologies described herein allow the child to still read the writtenwords, but then also watch the experience through a short movie clip oranimation as an outsider looking in. Same as in a pop-up context, or ARcontext, but the child •can interact with the main character by tappingon an inanimate object or screen to get some interaction from thecharacter, but the end result on the viewer is the same way you wouldtry to understand the situation by hearing a friend tell the story aboutthe situation they just went through without you actually being placedin that position yourself so you could begin to know how it feelsfirsthand, as it would relate to you.

Accordingly, in one aspect, disclosed herein are computer-implementedmethods of enhancing a publication comprising: receiving live video,captured by a user, comprising video of a publication, the publicationcomprising copyrighted content; identifying at least one first triggerin the live video; identifying one or more first three-dimensional,interactive media associated with the at least one first trigger andpertaining to the copyrighted content; presenting, via a first visualoutput device and to the user, the first three-dimensional, interactivemedia; identifying at least one second trigger in the firstthree-dimensional, interactive media; identifying one or more secondthree-dimensional, interactive media associated with the at least onesecond trigger and pertaining to the copyrighted content; andpresenting, via the first or a second visual output device and to theuser, the second three-dimensional, interactive media; whereby theuser's engagement with the copyrighted content of the publication isprogressively deepened and enriched. In some embodiments, thepublication is a printed publication. In other embodiments, thepublication is an e-book. In various embodiments, the publicationcomprises one or more selected from the group consisting of: a fictionnovel, a non-fiction novel, a children's book, a newspaper, a magazine,a comic book, a comic strip, a graphic novel, an encyclopedia, adictionary, a map, a calendar, a greeting card, a bookmark, an eventticket stub, and a poster. In some embodiments, the at least one firsttrigger comprises text, an image, a QR code, or a barcode associatedwith the publication. In other embodiments, the at least one firsttrigger comprises a three-dimensional object associated with thepublication. In some embodiments, the first three-dimensional,interactive media comprises a 360-degree video. In other embodiments,the first three-dimensional, interactive media comprises an augmentedreality display. In yet other embodiments, the first three-dimensional,interactive media comprises a multimedia game. In some embodiments, thefirst visual output device is a mobile device comprising a camera and adisplay screen. In some embodiments, the at least one second triggercomprises an interaction of the user with the first three-dimensional,interactive media. In further embodiments, the at least one secondtrigger comprises a duration of interaction. In some embodiments, thesecond three-dimensional, interactive media comprises virtual realityenvironment, a 360-degree video, or a multimedia game. In particularembodiments, the second three-dimensional, interactive media comprises avirtual reality environment. In further embodiments, the second visualoutput device is a virtual reality headset. In some embodiments, themethod further comprises determining a field of view of the user, thefield of view including at least a portion of the publication; modifyingan orientation of the one or more three-dimensional, interactive mediacontent to align with the field of view of the user; and presenting,with a visual output device and to the user, the one or morethree-dimensional, interactive media content, the orientation of whichhas been modified to align with the field of view of the user, such thatthe one or more three-dimensional, interactive media content ispresented as a virtual overlay in relation to the at least a portion ofthe publication. In some embodiments, the live video comprises aplurality of triggers. In further embodiments, the identifying one ormore first three-dimensional, interactive media associated with theplurality of first triggers comprises identifying one or more mediaassociated with a pre-determined combination of triggers. In someembodiments, the first three-dimensional, interactive media comprises aplurality of triggers. In further embodiments, the identifying one ormore second three-dimensional, interactive media associated with theplurality of second triggers comprises identifying one or more mediaassociated with a pre-determined combination of triggers. The firstthree-dimensional, interactive media and the second three-dimensional,interactive media, in some embodiments, pertain to the copyrightedcontent by way of comprising text, one or more graphics, one or morecharacters, story line, theme, one or more interactions, music, one ormore sound effects, spoken word audio relevant to the copyrightedcontent, or a combination thereof.

In another aspect, disclosed herein are computer-implemented systemscomprising: a digital processing device comprising: at least oneprocessor, a memory, a display, a camera, an operating system configuredto perform executable instructions, and a computer program includinginstructions executable by the digital processing device to create anapplication for enhancing a publication comprising: a first softwaremodule receiving live video captured by a user via the camera, the livevideo comprising video of a publication, the publication comprisingcopyrighted content; a second software module identifying at least onefirst trigger in the live video, identifying one or more firstthree-dimensional, interactive media associated with the at least onefirst trigger and pertaining to the copyrighted content, and presenting,via the display and to the user, the first three-dimensional,interactive media; and a third software module identifying at least onesecond trigger in the first three-dimensional, interactive media,identifying one or more second three-dimensional, interactive mediaassociated with the at least one second trigger and pertaining to thecopyrighted content, and presenting, via the display and to the user,the second three-dimensional, interactive media; whereby the user'sengagement with the copyrighted content of the publication isprogressively deepened and enriched. In some embodiments, thepublication is a printed publication. In other embodiments, thepublication is an e-book. In various embodiments, the publicationcomprises one or more selected from the group consisting of: a fictionnovel, a non-fiction novel, a children's book, a newspaper, a magazine,a comic book, a comic strip, a graphic novel, an encyclopedia, adictionary, a map, a calendar, a greeting card, a bookmark, an eventticket stub, and a poster. In some embodiments, the at least one firsttrigger comprises text, an image, a QR code, or a barcode associatedwith the publication. In other embodiments, the at least one firsttrigger comprises a three-dimensional object associated with thepublication. In some embodiments, the first three-dimensional,interactive media comprises a 360-degree video. In other embodiments,the first three-dimensional, interactive media comprises an augmentedreality display. In yet other embodiments, the first three-dimensional,interactive media comprises a multimedia game. In some embodiments, theat least one second trigger comprises an interaction of the user withthe first three-dimensional, interactive media. In further embodiments,the at least one second trigger comprises a duration of interaction. Insome embodiments, the second three-dimensional, interactive mediacomprises virtual reality environment, a 360-degree video, or amultimedia game. In particular embodiments, the secondthree-dimensional, interactive media comprises a virtual realityenvironment. In some embodiments, the application further comprises: afourth software module determining a field of view of the user, thefield of view including at least a portion of the publication, modifyingan orientation of the one or more three-dimensional, interactive mediacontent to align with the field of view of the user, and presenting, viathe display and to the user, the one or more three-dimensional,interactive media content, the orientation of which has been modified toalign with the field of view of the user, such that the one or morethree-dimensional, interactive media content is presented as a virtualoverlay in relation to the at least a portion of the publication. Insome embodiments, the live video comprises a plurality of triggers. Infurther embodiments, the software module identifying one or more firstthree-dimensional, interactive media associated with the plurality offirst triggers identifies one or more media associated with apre-determined combination of triggers. In some embodiments, the firstthree-dimensional, interactive media comprises a plurality of triggers.In further embodiments, the software module identifying one or moresecond three-dimensional, interactive media associated with theplurality of second triggers identifies one or more media associatedwith a pre-determined combination of triggers. The firstthree-dimensional, interactive media and the second three-dimensional,interactive media, in some embodiments, pertain to the copyrightedcontent by way of comprising text, one or more graphics, one or morecharacters, story line, theme, one or more interactions, music, one ormore sound effects, spoken word audio relevant to the copyrightedcontent, or a combination thereof.

In another aspect, disclosed herein are non-transitory computer-readablestorage media encoded with a computer program including instructionsexecutable by a processor to create an application for enhancing apublication comprising: a first software module receiving live videocaptured by a user via the camera of a mobile device, the live videocomprising video of a publication, the publication comprisingcopyrighted content; a second software module identifying at least onefirst trigger in the live video, identifying one or more firstthree-dimensional, interactive media associated with the at least onefirst trigger and pertaining to the copyrighted content, and presenting,via the display of the mobile device and to the user, the firstthree-dimensional, interactive media; and a third software moduleidentifying at least one second trigger in the first three-dimensional,interactive media, identifying one or more second three-dimensional,interactive media associated with the at least one second trigger andpertaining to the copyrighted content, and presenting, via the displayof the mobile device and to the user, the second three-dimensional,interactive media; whereby the user's engagement with the copyrightedcontent of the publication is progressively deepened and enriched. Insome embodiments, the publication is a printed publication. In otherembodiments, the publication is an e-book. In various embodiments, thepublication comprises one or more selected from the group consisting of:a fiction novel, a non-fiction novel, a children's book, a newspaper, amagazine, a comic book, a comic strip, a graphic novel, an encyclopedia,a dictionary, a map, a calendar, a greeting card, a bookmark, an eventticket stub, and a poster. In some embodiments, the at least one firsttrigger comprises text, an image, a QR code, or a barcode associatedwith the publication. In other embodiments, the at least one firsttrigger comprises a three-dimensional object associated with thepublication. In some embodiments, the first three-dimensional,interactive media comprises a 360-degree video. In other embodiments,the first three-dimensional, interactive media comprises an augmentedreality display. In yet other embodiments, the first three-dimensional,interactive media comprises a multimedia game. In some embodiments, theat least one second trigger comprises an interaction of the user withthe first three-dimensional, interactive media. In further embodiments,the at least one second trigger comprises a duration of interaction. Insome embodiments, the second three-dimensional, interactive mediacomprises virtual reality environment, a 360-degree video, or amultimedia game. In particular embodiments, the secondthree-dimensional, interactive media comprises a virtual realityenvironment. In some embodiments, the application further comprises: afourth software module determining a field of view of the user, thefield of view including at least a portion of the publication, modifyingan orientation of the one or more three-dimensional, interactive mediacontent to align with the field of view of the user, and presenting, viathe display and to the user, the one or more three-dimensional,interactive media content, the orientation of which has been modified toalign with the field of view of the user, such that the one or morethree-dimensional, interactive media content is presented as a virtualoverlay in relation to the at least a portion of the publication. Insome embodiments, the live video comprises a plurality of triggers. Infurther embodiments, the software module identifying one or more firstthree-dimensional, interactive media associated with the plurality offirst triggers identifies one or more media associated with apre-determined combination of triggers. In some embodiments, the firstthree-dimensional, interactive media comprises a plurality of triggers.In further embodiments, the software module identifying one or moresecond three-dimensional, interactive media associated with theplurality of second triggers identifies one or more media associatedwith a pre-determined combination of triggers. The firstthree-dimensional, interactive media and the second three-dimensional,interactive media, in some embodiments, pertain to the copyrightedcontent by way of comprising text, one or more graphics, one or morecharacters, story line, theme, one or more interactions, music, one ormore sound effects, spoken word audio relevant to the copyrightedcontent, or a combination thereof.

In another aspect, disclosed herein are methods of enhancing apublication, comprising: receiving, by a computing device of a user,live video comprising video of a publication; identifying at least onetrigger in the live video; identifying one or more media contentassociated with the at least one trigger; determining, by the computingdevice, a field of view of a user, the field of view including at leasta portion of the publication; modifying an orientation of the one ormore media content to align with the field of view of the user; andpresenting, with a visual output device and to the user, the one or moremedia content whose orientation has been modified to align with thefield of view of the user, such that the one or more media content ispresented as a virtual overlay in relation to the at least a portion ofthe publication. In some embodiments, the publication is a printedpublication. In other embodiments, the publication is an e-book. Invarious embodiments, the publication comprises one of: a fiction novel,a non-fiction novel, a children's book, a newspaper, a magazine, a comicbook, a comic strip, a graphic novel, an encyclopedia, a dictionary, amap, a calendar, a greeting card, a bookmark, an event ticket stub, anda poster. In some embodiments, the at least one trigger each comprisesone of: a QR code provided on the at least a portion of the publication,a barcode provided on the at least a portion of the publication, agraphical code provided on a sticker affixed to the at least a portionof the publication, a graphic on the at least a portion of thepublication, a keyword or key code on the at least a portion of thepublication, a cover image of the publication, a title on a cover of thepublication, author name on the cover of the publication, a stuffedcharacter associated with the publication that is positioned inproximity to the publication, a 3D-printed character associated with thepublication that is positioned in proximity to the publication, and a3D-printed object associated with the publication that is positioned inproximity to the publication. In some embodiments, the live videocomprises at least 2 triggers, at least 3 triggers, at least 4 triggers,at least 5 triggers, at least 6 triggers, at least 7 triggers, at least8 triggers, at least 9 triggers, or at least 10 triggers. In furtherembodiments, the identifying one or more media content associated withthe at least one trigger comprises identifying one or more media contentassociated with a pre-determined combination of triggers. In stillfurther embodiments, the combination of triggers comprises the identityof each trigger and the sequence of the identification. In someembodiments, the one or more media content comprises at least one of: a3D still graphic, a 3D animated graphic, and a 3D interactive animatedgraphic. In further embodiments, each of the 3D still graphic, the 3Danimated graphic, or the 3D interactive animated graphic comprises atleast one of: a graphic representation of one or more charactersassociated with the publication, a non-fictional geographical locationassociated with the publication, a fictional geographical locationassociated with the publication, a room associated with the publication,a building associated with the publication, a scene described in thepublication, and one or more items described in the publication. Instill further embodiments, each of the 3D still graphic, the 3D animatedgraphic, or the 3D interactive animated graphic is presented from thepoint of view of a character associated with the publication. In someembodiments, presenting the one or more media content comprisespresenting, with the visual output device and to the user, the one ormore media content whose orientation has been modified to align with thefield of view of the user, such that the one or more media content is atleast one of presented as a virtual overlay above a cover of thepublication, presented as a virtual overlay on a cover of thepublication, presented as a virtual overlay beside the publication,presented as a virtual overlay above a page of the publication,presented as a virtual overlay on a page of the publication, presentedas a virtual overlay in front of a surface of the publication, presentedas a virtual overlay in front of a binding of the publication, presentedas a virtual overlay above a 2D graphic on a portion of the publication,or presented as a virtual overlay on a 2D graphic on a portion of thepublication. In some embodiments, the visual output device comprises oneof: a VR headset, an AR headset, a set of AR glasses, a tablet computer,a set of bionic contact lenses, a smart phone adapted as part of a VRheadset, and a smart phone adapted as part of an AR system. In someembodiments, the visual output device is communicatively coupled withthe computing device of the user. In some embodiments, the visual outputdevice comprises a second camera that captures one or more images ofeyes of the user as the first camera captures the one or more images ofthe at least a portion of the publication, wherein determining, with thecomputer system, a field of view of a user comprises determining, withthe computer system, a field of view of a user based on an analysis ofthe captured one or more images of the eyes of the user. In someembodiments, the identifying at least one trigger in the live video isperformed at the computing device of the user or at a servercommunicatively coupled to the computing device of the user. In someembodiments, the identifying one or more media content associated withthe at least one trigger is performed at the computing device of theuser or at a server communicatively coupled to the computing device ofthe user. In some embodiments, the modifying an orientation of the oneor more media content to align with the field of view of the user isperformed at the computing device of the user or at a servercommunicatively coupled to the computing device of the user. In someembodiments, the computing device comprises a mobile phone or a tabletcomputer.

In another aspect, disclosed herein are computer-implemented systemscomprising a computing device comprising at least one processor, anoperating system configured to perform executable instructions, amemory, a camera, and a computer program including instructionsexecutable by the at least one processor to create an applicationcomprising: a software module receiving from the camera live videocomprising video of a publication; a software module identifying atleast one trigger in the live video; a software module identifying oneor more media content associated with the at least one trigger; asoftware module determining a field of view of a user, the field of viewincluding at least a portion of the publication; a software modulemodifying an orientation of the one or more media content to align withthe field of view of the user; and a software module presenting, via avisual output device and to the user, the one or more media contentwhose orientation has been modified to align with the field of view ofthe user, such that the one or more media content is presented as avirtual overlay in relation to the at least a portion of thepublication. In some embodiments, the publication is a printedpublication. In other embodiments, the publication is an e-book. Invarious embodiments, the publication comprises one of: a fiction novel,a non-fiction novel, a children's book, a newspaper, a magazine, a comicbook, a comic strip, a graphic novel, an encyclopedia, a dictionary, amap, a calendar, a greeting card, a bookmark, an event ticket stub, anda poster. In some embodiments, the at least one trigger each comprisesone of: a QR code provided on the at least a portion of the publication,a barcode provided on the at least a portion of the publication, agraphical code provided on a sticker affixed to the at least a portionof the publication, a graphic on the at least a portion of thepublication, a keyword or key code on the at least a portion of thepublication, a cover image of the publication, a title on a cover of thepublication, author name on the cover of the publication, a stuffedcharacter associated with the publication that is positioned inproximity to the publication, a 3D-printed character associated with thepublication that is positioned in proximity to the publication, and a3D-printed object associated with the publication that is positioned inproximity to the publication. In some embodiments, the live videocomprises at least 2 triggers, at least 3 triggers, at least 4 triggers,at least 5 triggers, at least 6 triggers, at least 7 triggers, at least8 triggers, at least 9 triggers, or at least 10 triggers. In furtherembodiments, the software module identifying one or more media contentassociated with the at least one trigger identifies one or more mediacontent associated with a pre-determined combination of triggers. Instill further embodiments, the combination of triggers comprises theidentity of each trigger and the sequence of the identification. In someembodiments, the one or more media content comprises at least one of: 3Dstill graphic, a 3D animated graphic, and a 3D interactive animatedgraphic. In further embodiments, each of the 3D still graphic, the 3Danimated graphic, or the 3D interactive animated graphic comprises atleast one of: a graphic representation of one or more charactersassociated with the publication, a non-fictional geographical locationassociated with the publication, a fictional geographical locationassociated with the publication, a room associated with the publication,a building associated with the publication, a scene described in thepublication, and one or more items described in the publication. Instill further embodiments, each of the 3D still graphic, the 3D animatedgraphic, or the 3D interactive animated graphic is presented from thepoint of view of a character associated with the publication. In someembodiments, presenting the one or more media content comprisespresenting, with the visual output device and to the user, the one ormore media content whose orientation has been modified to align with thefield of view of the user, such that the one or more media content is atleast one of presented as a virtual overlay above a cover of thepublication, presented as a virtual overlay on a cover of thepublication, presented as a virtual overlay beside the publication,presented as a virtual overlay above a page of the publication,presented as a virtual overlay on a page of the publication, presentedas a virtual overlay in front of a surface of the publication, presentedas a virtual overlay in front of a binding of the publication, presentedas a virtual overlay above a 2D graphic on a portion of the publication,or presented as a virtual overlay on a 2D graphic on a portion of thepublication. In some embodiments, the visual output device comprises oneof: a VR headset, an AR headset, a set of AR glasses, a tablet computer,a set of bionic contact lenses, a smart phone adapted as part of a VRheadset, and a smart phone adapted as part of an AR system. In someembodiments, the visual output device is communicatively coupled withthe computing device of the user. In some embodiments, the visual outputdevice comprises a second camera that captures one or more images ofeyes of the user as the first camera captures the one or more images ofthe at least a portion of the publication, wherein determining, with thecomputer system, a field of view of a user comprises determining, withthe computer system, a field of view of a user based on an analysis ofthe captured one or more images of the eyes of the user. In someembodiments, the computing device comprises a mobile phone or a tabletcomputer.

In another aspect, disclosed herein are non-transitory computer-readablestorage media encoded with a computer program including instructionsexecutable by a processor to create an application comprising: asoftware module receiving, via a camera associated with a computingdevice of a user, live video comprising video of a publication; asoftware module identifying at least one trigger in the live video; asoftware module identifying one or more media content associated withthe at least one trigger; a software module determining a field of viewof a user, the field of view including at least a portion of thepublication; a software module modifying an orientation of the one ormore media content to align with the field of view of the user; and asoftware module presenting, via a visual output device and to the user,the one or more media content whose orientation has been modified toalign with the field of view of the user, such that the one or moremedia content is presented as a virtual overlay in relation to the atleast a portion of the publication. In some embodiments, the publicationis a printed publication. In other embodiments, the publication is ane-book. In various embodiments, the publication comprises one of: afiction novel, a nonfiction novel, a children's book, a newspaper, amagazine, a comic book, a comic strip, a graphic novel, an encyclopedia,a dictionary, a map, a calendar, a greeting card, a bookmark, an eventticket stub, and a poster. In various embodiments, the at least onetrigger each comprises one of: a QR code provided on the at least aportion of the publication, a barcode provided on the at least a portionof the publication, a graphical code provided on a sticker affixed tothe at least a portion of the publication, a graphic on the at least aportion of the publication, a keyword or key code on the at least aportion of the publication, a cover image of the publication, a title ona cover of the publication, author name on the cover of the publication,a stuffed character associated with the publication that is positionedin proximity to the publication, a 3D-printed character associated withthe publication that is positioned in proximity to the publication, anda 3D-printed object associated with the publication that is positionedin proximity to the publication. In some embodiments, the live videocomprises at least 2 triggers, at least 3 triggers, at least 4 triggers,at least 5 triggers, at least 6 triggers, at least 7 triggers, at least8 triggers, at least 9 triggers, or at least 10 triggers. In furtherembodiments, the software module identifying one or more media contentassociated with the at least one trigger identifies one or more mediacontent associated with a pre-determined combination of triggers. Instill further embodiments, the combination of triggers comprises theidentity of each trigger and the sequence of the identification. In someembodiments, the one or more media content comprises at least one of: a3D still graphic, a 3D animated graphic, and a 3D interactive animatedgraphic. In further embodiments, each of the 3D still graphic, the 3Danimated graphic, or the 3D interactive animated graphic comprises atleast one of: a graphic representation of one or more charactersassociated with the publication, a non-fictional geographical locationassociated with the publication, a fictional geographical locationassociated with the publication, a room associated with the publication,a building associated with the publication, a scene described in thepublication, and one or more items described in the publication. Instill further embodiments, each of the 3D still graphic, the 3D animatedgraphic, or the 3D interactive animated graphic is presented from thepoint of view of a character associated with the publication. In someembodiments, presenting the one or more media content comprisespresenting, with the visual output device and to the user, the one ormore media content whose orientation has been modified to align with thefield of view of the user, such that the one or more media content is atleast one of presented as a virtual overlay above a cover of thepublication, presented as a virtual overlay on a cover of thepublication, presented as a virtual overlay beside the publication,presented as a virtual overlay above a page of the publication,presented as a virtual overlay on a page of the publication, presentedas a virtual overlay in front of a surface of the publication, presentedas a virtual overlay in front of a binding of the publication, presentedas a virtual overlay above a 2D graphic on a portion of the publication,or presented as a virtual overlay on a 2D graphic on a portion of thepublication. In various embodiments, the visual output device comprisesone of: a VR headset, an AR headset, a set of AR glasses, a tabletcomputer, a set of bionic contact lenses, a smart phone adapted as partof a VR headset, and a smart phone adapted as part of an AR system. Insome embodiments, the visual output device is communicatively coupledwith the computing device of the user. In some embodiments, the visualoutput device comprises a second camera that captures one or more imagesof eyes of the user as the first camera captures the one or more imagesof the at least a portion of the publication, wherein determining, withthe computer system, a field of view of a user comprises determining,with the computer system, a field of view of a user based on an analysisof the captured one or more images of the eyes of the user. In someembodiments, the computing device comprises a mobile phone or a tabletcomputer.

In another aspect, disclosed herein are methods, comprising: capturing,with a first camera, one or more images of at least a portion of aprinted publication; analyzing, with a computing system, the capturedone or more images of the at least a portion of the printed publicationto determine whether the captured one or more images contain one or moretriggers; based on a determination that the captured one or more imagescontain at least one trigger of the one or more triggers, identifying,with the computing system, one or more media content of a plurality ofmedia content that are associated with the at least one trigger;determining, with the computer system, a field of view of a user, thefield of view including at least a portion of the printed publication;modifying, with the computer system, an orientation of the one or moremedia content to align with the field of view of the user; andpresenting, with a visual output device and to the user, the one or moremedia content whose orientation has been modified to align with thefield of view of the user, such that the one or more media content ispresented as a virtual overlay in relation to the at least a portion ofthe printed publication. In some embodiments, the visual output devicecomprises one of a VR headset, an AR headset, a set of AR glasses, atablet computer, a set of bionic contact lenses, a smart phone adaptedas part of a VR headset, or a smart phone adapted as part of an ARsystem. In some embodiments, the visual output device comprises thefirst camera. In some embodiments, the first camera is external to thevisual output device, while being communicatively coupled to the visualoutput device. In some embodiments, the visual output device comprises asecond camera that captures one or more images of eyes of the user asthe first camera captures the one or more images of the at least aportion of the printed publication, wherein determining, with thecomputer system, a field of view of a user comprises determining, withthe computer system, a field of view of a user based on an analysis ofthe captured one or more images of the eyes of the user. In someembodiments, the printed publication comprises one of a fiction novel, anon-fiction novel, a children's book, a newspaper, a magazine, a comicbook, a comic strip, a graphic novel, an encyclopedia, a dictionary, amap, a calendar, a greeting card, a bookmark, a venue or event ticketstub, or a poster. In some embodiments, the at least one trigger eachcomprises one of a QR code provided on the at least a portion of thepublication, a barcode provided on the at least a portion of thepublication, a graphical code provided on a sticker affixed to the atleast a portion of the publication, a graphic printed on the at least aportion of the publication, a keyword or key code printed on the atleast a portion of the publication, a cover image of the printedpublication, a title printed on a cover of the printed publication,author name printed on the cover of the printed publication, a stuffedcharacter associated with the printed publication that is positioned inproximity to the printed publication, a 3D-printed character associatedwith the printed publication that is positioned in proximity to theprinted publication, or a 3D-printed object associated with the printedpublication that is positioned in proximity to the printed publication.In some embodiments, the one or more media content comprises at leastone of a 3D still graphic, a 3D animated graphic, or a 3D interactiveanimated graphic. In further embodiments, each of the 3D still graphic,the 3D animated graphic, or the 3D interactive animated graphiccomprises at least one of a graphic representation of one or morecharacters associated with the printed publication, a non-fictionalgeographical location associated with the printed publication, afictional geographical location associated with the printed publication,a room associated with the printed publication, a building associatedwith the printed publication, a scene described in the printedpublication, a scene inspired by the scene described in the printedpublication, one or more items described in the printed publication, orone or more items inspired by the one or more items described in theprinted publication. In some embodiments, presenting the one or moremedia content comprises presenting, with the visual output device and tothe user, the one or more media content whose orientation has beenmodified to align with the field of view of the user, such that the oneor more media content is at least one of presented as a virtual overlayabove a cover of the printed publication, presented as a virtual overlayon a cover of the printed publication, presented as a virtual overlaybeside the printed publication, presented as a virtual overlay above apage of the printed publication, presented as a virtual overlay on apage of the printed publication, presented as a virtual overlay in frontof a surface of the printed publication, presented as a virtual overlayin front of a binding of the printed publication, presented as a virtualoverlay above a 2D graphic printed on a portion of the printedpublication, or presented as a virtual overlay on a 2D graphic printedon a portion of the printed publication. In some embodiments, thecomputing system comprises at least one of a server computer, a cloudcomputing system, or a computing system of the visual output device.

In another aspect, disclosed herein are apparatus, comprising: a firstcamera; a display device; at least one processor; and a non-transitorycomputer readable medium, the non-transitory computer readable mediumhaving stored thereon computer software comprising a set of instructionsthat, when executed by the at least one processors, causes the apparatusto: capture, with the first camera, one or more images of at least aportion of a printed publication; analyze the captured one or moreimages of the at least a portion of the printed publication to determinewhether the captured one or more images contain one or more triggers;based on a determination that the captured one or more images contain atleast one trigger, identify one or more media content of a plurality ofmedia content that are associated with the at least one trigger;determine a field of view of a user, the field of view including atleast a portion of the printed publication; modify an orientation of theone or more media content to align with the field of view of the user;and present, with the display device and to the user, the one or moremedia content whose orientation has been modified to align with thefield of view of the user, such that the one or more media content ispresented as a virtual overlay in relation to the at least a portion ofthe printed publication. In some embodiments, the apparatus comprisesone of a VR headset, an AR headset, a set of AR glasses, a tabletcomputer, a set of bionic contact lenses, a smart phone adapted as partof a VR headset, or a smart phone adapted as part of an AR system. Insome embodiments, further comprises: a second camera; wherein the set ofinstructions, when executed by the one or more processors, furthercauses the apparatus to: capture, with the second camera, one or moreimages of eyes of the user as the first camera captures the one or moreimages of the at least a portion of the printed publication; whereindetermining a field of view of a user comprises determining a field ofview of a user based on an analysis of the captured one or more imagesof the eyes of the user. In some embodiments, the printed publicationcomprises one of a fiction novel, a non-fiction novel, a children'sbook, a newspaper, a magazine, a comic book, a comic strip, a graphicnovel, an encyclopedia, a dictionary, a map, a calendar, a greetingcard, a bookmark, a venue or event ticket stub, or a poster. In someembodiments, the at least one trigger each comprises one of a QR codeprovided on the at least a portion of the publication, a barcodeprovided on the at least a portion of the publication, a graphical codeprovided on a sticker affixed to the at least a portion of thepublication, a graphic printed on the at least a portion of thepublication, a keyword or key code printed on the at least a portion ofthe publication, a cover image of the printed publication, a titleprinted on a cover of the printed publication, author name printed onthe cover of the printed publication, a stuffed character associatedwith the printed publication that is positioned in proximity to theprinted publication, a 3D-printed character associated with the printedpublication that is positioned in proximity to the printed publication,or a 3D-printed object associated with the printed publication that ispositioned in proximity to the printed publication. In some embodiments,the one or more media content comprises at least one of a 3D stillgraphic, a 3D animated graphic, or a 3D interactive animated graphic. Insome embodiments, each of the 3D still graphic, the 3D animated graphic,or the 3D interactive animated graphic comprises at least one of agraphic representation of one or more characters associated with theprinted publication, a non-fictional geographical location associatedwith the printed publication, a fictional geographical locationassociated with the printed publication, a room associated with theprinted publication, a building associated with the printed publication,a scene described in the printed publication, a scene inspired by thescene described in the printed publication, one or more items describedin the printed publication, or one or more items inspired by the one ormore items described in the printed publication. In some embodiments,presenting the one or more media content comprises presenting, with thedisplay device and to the user, the one or more media content whoseorientation has been modified to align with the field of view of theuser, such that the one or more media content is at least one ofpresented as a virtual overlay above a cover of the printed publication,presented as a virtual overlay on a cover of the printed publication,presented as a virtual overlay beside the printed publication, presentedas a virtual overlay above a page of the printed publication, presentedas a virtual overlay on a page of the printed publication, presented asa virtual overlay in front of a surface of the printed publication,presented as a virtual overlay in front of a binding of the printedpublication, presented as a virtual overlay above a 2D graphic printedon a portion of the printed publication, or presented as a virtualoverlay on a 2D graphic printed on a portion of the printed publication.

In another aspect, disclosed herein are systems, comprising: an imagecapture device, comprising: a first camera; at least one firstprocessor; and a first non-transitory computer readable medium, thefirst non-transitory computer readable medium having stored thereoncomputer software comprising a first set of instructions that, whenexecuted by the at least one first processors, causes the image capturedevice to: capture, with the first camera, one or more images of atleast a portion of a printed publication; and send the captured one ormore images to a computing system; the computing system, comprising: atleast one second processor; and a second non-transitory computerreadable medium, the second non-transitory computer readable mediumhaving stored thereon computer software comprising a second set ofinstructions that, when executed by the at least one second processors,causes the computing system to: receive the captured one or more imagesfrom the image capture device; analyze the captured one or more imagesof the at least a portion of the printed publication to determinewhether the captured one or more images contain one or more triggers;based on a determination that the captured one or more images contain atleast one trigger, identify one or more media content of a plurality ofmedia content that are associated with the at least one trigger;determine a field of view of a user, the field of view including atleast a portion of the printed publication; modify an orientation of theone or more media content to align with the field of view of the user;and send the one or more media content whose orientation has beenmodified to align with the field of view of the user to a visual outputdevice; and the visual output device, comprising: a display device; atleast one third processor; and a third non-transitory computer readablemedium, the third non-transitory computer readable medium having storedthereon computer software comprising a third set of instructions that,when executed by the at least one third processors, causes the visualoutput device to: receive the one or more media content whoseorientation has been modified to align with the field of view of theuser from the computing system; and present, with the display device andto the user, the one or more media content whose orientation has beenmodified to align with the field of view of the user, such that the oneor more media content is presented as a virtual overlay in relation tothe at least a portion of the printed publication. In some embodiments,the visual output device comprises one of a VR headset, an AR headset, aset of AR glasses, a tablet computer, a set of bionic contact lenses, asmart phone adapted as part of a VR headset, or a smart phone adapted aspart of an AR system. In some embodiments, the visual output devicecomprises the image capture device. In some embodiments, the visualoutput device comprises a second camera that captures one or more imagesof eyes of the user as the first camera captures the one or more imagesof the at least a portion of the printed publication, whereindetermining, with the computer system, a field of view of a usercomprises determining, with the computer system, a field of view of auser based on an analysis of the captured one or more images of the eyesof the user. In some embodiments, the printed publication comprises oneof a fiction novel, a non-fiction novel, a children's book, a newspaper,a magazine, a comic book, a comic strip, a graphic novel, anencyclopedia, a dictionary, a map, a calendar, a greeting card, abookmark, a venue or event ticket stub, or a poster. In someembodiments, the at least one trigger each comprises one of a QR codeprovided on the at least a portion of the publication, a barcodeprovided on the at least a portion of the publication, a graphical codeprovided on a sticker affixed to the at least a portion of thepublication, a graphic printed on the at least a portion of thepublication, a keyword or key code printed on the at least a portion ofthe publication, a cover image of the printed publication, a titleprinted on a cover of the printed publication, author name printed onthe cover of the printed publication, a stuffed character associatedwith the printed publication that is positioned in proximity to theprinted publication, a 3D-printed character associated with the printedpublication that is positioned in proximity to the printed publication,or a 3D-printed object associated with the printed publication that ispositioned in proximity to the printed publication. In some embodiments,the one or more media content comprises at least one of a 3D stillgraphic, a 3D animated graphic, or a 3D interactive animated graphic. Infurther embodiments, each of the 3D still graphic, the 3D animatedgraphic, or the 3D interactive animated graphic comprises at least oneof a graphic representation of one or more characters associated withthe printed publication, a non-fictional geographical locationassociated with the printed publication, a fictional geographicallocation associated with the printed publication, a room associated withthe printed publication, a building associated with the printedpublication, a scene described in the printed publication, a sceneinspired by the scene described in the printed publication, one or moreitems described in the printed publication, or one or more itemsinspired by the one or more items described in the printed publication.In some embodiments, presenting the one or more media content comprisespresenting, with the visual output device and to the user, the one ormore media content whose orientation has been modified to align with thefield of view of the user, such that the one or more media content is atleast one of presented as a virtual overlay above a cover of the printedpublication, presented as a virtual overlay on a cover of the printedpublication, presented as a virtual overlay beside the printedpublication, presented as a virtual overlay above a page of the printedpublication, presented as a virtual overlay on a page of the printedpublication, presented as a virtual overlay in front of a surface of theprinted publication, presented as a virtual overlay in front of abinding of the printed publication, presented as a virtual overlay abovea 2D graphic printed on a portion of the printed publication, orpresented as a virtual overlay on a 2D graphic printed on a portion ofthe printed publication. In some embodiments, the computing systemcomprises at least one of a server computer, a cloud computing system,or a computing system of the visual output device.

In another aspect, disclosed herein are methods, comprising:establishing communications between a visual output device and at leastone trigger of one or more non-visual triggers associated with a printedpublication; in response to establishing communications between thevisual output device and the at least one trigger, identifying, with thecomputing system, one or more media content of a plurality of mediacontent that are associated with the at least one trigger; determining,with the computer system, a field of view of a user, the field of viewincluding at least a portion of the printed publication; modifying, withthe computer system, an orientation of the one or more media content toalign with the field of view of the user; and presenting, with thevisual output device and to the user, the one or more media contentwhose orientation has been modified to align with the field of view ofthe user, such that the one or more media content is presented as avirtual overlay in relation to the at least a portion of the printedpublication. In some embodiments, establishing communications betweenthe visual output device and the at least one trigger comprises:determining, with the computing system, whether the visual output deviceis within communications range of the one or more non-visual triggersassociated with the printed publication; and based on a determinationthat the visual output device is within communications range of the atleast one trigger of the one or more non-visual triggers, establishingcommunications between the visual output device and the at least onetrigger. In some embodiments, the visual output device comprises one ofa VR headset, an AR headset, a set of AR glasses, a tablet computer, aset of bionic contact lenses, a smart phone adapted as part of a VRheadset, or a smart phone adapted as part of an AR system. In someembodiments, the visual output device comprises a camera that capturesone or more images of eyes of the user as the user is looking toward theat least a portion of the printed publication, wherein determining, withthe computer system, a field of view of a user comprises determining,with the computer system, a field of view of a user based on an analysisof the captured one or more images of the eyes of the user. In someembodiments, the printed publication comprises one of a fiction novel, anon-fiction novel, a children's book, a newspaper, a magazine, a comicbook, a comic strip, a graphic novel, an encyclopedia, a dictionary, amap, a calendar, a greeting card, a bookmark, a venue or event ticketstub, or a poster. In some embodiments, the at least one trigger eachcomprises one of a RFID tag, a BLE tag, a NFC tag, or a RTLS tag. Insome embodiments, the one or more media content comprises at least oneof a 3D still graphic, a 3D animated graphic, or a 3D interactiveanimated graphic. In further embodiments, each of the 3D still graphic,the 3D animated graphic, or the 3D interactive animated graphiccomprises at least one of a graphic representation of one or morecharacters associated with the printed publication, a non-fictionalgeographical location associated with the printed publication, afictional geographical location associated with the printed publication,a room associated with the printed publication, a building associatedwith the printed publication, a scene described in the printedpublication, a scene inspired by the scene described in the printedpublication, one or more items described in the printed publication, orone or more items inspired by the one or more items described in theprinted publication. In some embodiments, presenting the one or moremedia content comprises presenting, with the visual output device and tothe user, the one or more media content whose orientation has beenmodified to align with the field of view of the user, such that the oneor more media content is at least one of presented as a virtual overlayabove a cover of the printed publication, presented as a virtual overlayon a cover of the printed publication, presented as a virtual overlaybeside the printed publication, presented as a virtual overlay above apage of the printed publication, presented as a virtual overlay on apage of the printed publication, presented as a virtual overlay in frontof a surface of the printed publication, presented as a virtual overlayin front of a binding of the printed publication, presented as a virtualoverlay above a 2D graphic printed on a portion of the printedpublication, or presented as a virtual overlay on a 2D graphic printedon a portion of the printed publication. In some embodiments, thecomputing system comprises at least one of a server computer, a cloudcomputing system, or a computing system of the visual output device.

In another aspect, disclosed herein are apparatus, comprising: awireless communications device; a display device; at least oneprocessor; and a non-transitory computer readable medium, thenon-transitory computer readable medium having stored thereon computersoftware comprising a set of instructions that, when executed by the atleast one processors, causes the apparatus to: establish, with thewireless communications device, communications between a visual outputdevice and at least one trigger of one or more non-visual triggersassociated with a printed publication; in response to establishingcommunications between the visual output device and the at least onetrigger, identify one or more media content of a plurality of mediacontent that are associated with the at least one trigger; determine afield of view of a user, the field of view including at least a portionof the printed publication; modify an orientation of the one or moremedia content to align with the field of view of the user; and present,with the display device and to the user, the one or more media contentwhose orientation has been modified to align with the field of view ofthe user, such that the one or more media content is presented as avirtual overlay in relation to the at least a portion of the printedpublication. In some embodiments, the apparatus comprises one of a VRheadset, an AR headset, a set of AR glasses, a tablet computer, a set ofbionic contact lenses, a smart phone adapted as part of a VR headset, ora smart phone adapted as part of an AR system. In some embodiments, theapparatus further comprises a camera that captures one or more images ofeyes of the user as the user is looking toward the at least a portion ofthe printed publication, wherein determining, with the computer system,a field of view of a user comprises determining, with the computersystem, a field of view of a user based on an analysis of the capturedone or more images of the eyes of the user. In some embodiments, theprinted publication comprises one of a fiction novel, a non-fictionnovel, a children's book, a newspaper, a magazine, a comic book, a comicstrip, a graphic novel, an encyclopedia, a dictionary, a map, acalendar, a greeting card, a bookmark, a venue or event ticket stub, ora poster. In some embodiments, the at least one trigger each comprisesone of a RFID tag, a BLE tag, a NFC tag, or a RTLS tag. In someembodiments, the one or more media content comprises at least one of a3D still graphic, a 3D animated graphic, or a 3D interactive animatedgraphic.

In another aspect, disclosed herein are systems, comprising: a wirelesscommunication device, comprising: a transceiver; at least one firstprocessor; and a first non-transitory computer readable medium, thefirst non-transitory computer readable medium having stored thereoncomputer software comprising a first set of instructions that, whenexecuted by the at least one first processors, causes the image capturedevice to: establish communications between a visual output device andat least one trigger of one or more non-visual triggers associated witha printed publication; the computing system, comprising: at least onesecond processor; and a second non-transitory computer readable medium,the second non-transitory computer readable medium having stored thereoncomputer software comprising a second set of instructions that, whenexecuted by the at least one second processors, causes the computingsystem to: in response to establishing communications between the visualoutput device and the at least one trigger, identify one or more mediacontent of a plurality of media content that are associated with the atleast one trigger; determine a field of view of a user, the field ofview including at least a portion of the printed publication; modify anorientation of the one or more media content to align with the field ofview of the user; and send the one or more media content whoseorientation has been modified to align with the field of view of theuser to a visual output device; and the visual output device,comprising: a display device; at least one third processor; and a thirdnon-transitory computer readable medium, the third non-transitorycomputer readable medium having stored thereon computer softwarecomprising a third set of instructions that, when executed by the atleast one third processors, causes the visual output device to: receivethe one or more media content whose orientation has been modified toalign with the field of view of the user from the computing system; andpresent, with the display device and to the user, the one or more mediacontent whose orientation has been modified to align with the field ofview of the user, such that the one or more media content is presentedas a virtual overlay in relation to the at least a portion of theprinted publication.

In another aspect, disclosed herein are methods, comprising: presenting,with a visual output device and to a user, one or more media content,such that the one or more media content is presented as a virtualoverlay in relation to at least a portion of a physical object;determining, with a computing system, how the one or more media contentshould be presented to the user; based on a determination to present theone or more media content as a virtual reality presentation, and basedon a determination that the one or more media content is not alreadybeing presented as a virtual reality presentation, shifting, with thecomputing system, the presentation of the one or more media content to avirtual reality presentation, and presenting, with the visual outputdevice and to the user, one or more second media content, the one ormore second media content comprising virtual reality content; and basedon a determination to present the one or more media content as a mixedreality presentation, and based on a determination that the one or moremedia content is not already being presented as a mixed realitypresentation, shifting, with the computing system, the presentation ofthe one or more media content to a mixed reality presentation, andpresenting, with the visual output device and to the user, one or morethird media content, the one or more third media content comprisingmixed reality content. In some embodiments, the visual output devicecomprises one of a VR headset, an AR headset, a set of AR glasses, atablet computer, a set of bionic contact lenses, a smart phone adaptedas part of a VR headset, or a smart phone adapted as part of an ARsystem. In some embodiments, the physical object comprises one of afiction novel, a non-fiction novel, a children's book, a newspaper, amagazine, a comic book, a comic strip, a graphic novel, an encyclopedia,a dictionary, a map, a calendar, a greeting card, a bookmark, a venue orevent ticket stub, a poster, a Christmas tree bulb, a Christmas treeornament, a Hanukkah menorah, a gravestone, a sentimental artifact orornament associated with the user, or a sentimental artifact or ornamentassociated with another individual who is associated with the user. Insome embodiments, the one or more media content comprises at least oneof a 3D still graphic, a 3D animated graphic, or a 3D interactiveanimated graphic. In some embodiments, the method further comprises atleast one of: detecting, with one or more sensors, one or more physicalcharacteristics of the user in response to stimuli from presentation ofmedia content; or receiving, with the computing system, a user inputindicating selection of how the media content should be presented;wherein determining, with the computing system, how the one or moremedia content should be presented to the user is based on at least oneof analysis of the detected one or more physical characteristics of theuser or the received user input. In some embodiments, the method furthercomprises: based on a determination to present the one or more mediacontent as an augmented reality presentation, and based on adetermination that the one or more media content is not already beingpresented as an augmented reality presentation, shifting, with thecomputing system, the presentation of the one or more media content toan augmented reality presentation, and presenting, with the visualoutput device and to the user, one or more fourth media content, the oneor more fourth media content comprising augmented reality content.

In another aspect, disclosed herein are apparatus, comprising: a displaydevice; at least one processor; and a non-transitory computer readablemedium, the non-transitory computer readable medium having storedthereon computer software comprising a set of instructions that, whenexecuted by the at least one processors, causes the apparatus to:present, with the display device and to a user, one or more mediacontent, such that the one or more media content is presented as avirtual overlay in relation to at least a portion of a physical object;determine how the one or more media content should be presented to theuser; based on a determination to present the one or more media contentas a virtual reality presentation, and based on a determination that theone or more media content is not already being presented as a virtualreality presentation, shift the presentation of the one or more mediacontent to a virtual reality presentation, and present, with the displaydevice and to the user, one or more second media content, the one ormore second media content comprising virtual reality content; and basedon a determination to present the one or more media content as a mixedreality presentation, and based on a determination that the one or moremedia content is not already being presented as a mixed realitypresentation, shift the presentation of the one or more media content toa mixed reality presentation, and present, with the display device andto the user, one or more third media content, the one or more thirdmedia content comprising mixed reality content. In some embodiments, theapparatus comprises one of a VR headset, an AR headset, a set of ARglasses, a tablet computer, a set of bionic contact lenses, a smartphone adapted as part of a VR headset, or a smart phone adapted as partof an AR system. In some embodiments, the physical object comprises oneof a fiction novel, a non-fiction novel, a children's book, a newspaper,a magazine, a comic book, a comic strip, a graphic novel, anencyclopedia, a dictionary, a map, a calendar, a greeting card, abookmark, a venue or event ticket stub, a poster, a Christmas tree bulb,a Christmas tree ornament, a Hanukkah menorah, a gravestone, asentimental artifact or ornament associated with the user, or asentimental artifact or ornament associated with another individual whois associated with the user. In some embodiments, the one or more mediacontent comprises at least one of a 3D still graphic, a 3D animatedgraphic, or a 3D interactive animated graphic. In some embodiments, theapparatus further comprises at least one of: one or more sensors; or oneor more user input devices; wherein the set of instructions, whenexecuted by the one or more processors, further causes the apparatus toperform at least one of: detecting, with the one or more sensors, one ormore physical characteristics of the user in response to stimuli frompresentation of media content; or receiving, from the one or more userinput devices, a user input indicating selection of how the mediacontent should be presented; wherein determining how the one or moremedia content should be presented to the user is based on at least oneof analysis of the detected one or more physical characteristics of theuser or the received user input. In further embodiments, the set ofinstructions, when executed by the one or more processors, furthercauses the apparatus to: based on a determination to present the one ormore media content as an augmented reality presentation, and based on adetermination that the one or more media content is not already beingpresented as an augmented reality presentation, shift the presentationof the one or more media content to an augmented reality presentation,and present, with the display device and to the user, one or more fourthmedia content, the one or more fourth media content comprising augmentedreality content.

In another aspect, disclosed herein are systems, comprising: a visualoutput device, comprising: a display device; at least one firstprocessor; and a first non-transitory computer readable medium, thefirst non-transitory computer readable medium having stored thereoncomputer software comprising a first set of instructions that, whenexecuted by the at least one first processors, causes the visual outputdevice to: present, with the display device and to a user, one or moremedia content, such that the one or more media content is presented as avirtual overlay in relation to at least a portion of a physical object;a computing system, comprising: at least one second processor; and asecond non-transitory computer readable medium, the secondnon-transitory computer readable medium having stored thereon computersoftware comprising a second set of instructions that, when executed bythe at least one second processors, causes the computing system to:determine how the one or more media content should be presented to theuser; based on a determination to present the one or more media contentas a virtual reality presentation, and based on a determination that theone or more media content is not already being presented as a virtualreality presentation, shift the presentation of the one or more mediacontent to a virtual reality presentation; and based on a determinationto present the one or more media content as a mixed realitypresentation, and based on a determination that the one or more mediacontent is not already being presented as a mixed reality presentation,shift the presentation of the one or more media content to a mixedreality presentation; wherein the first set of instructions that, whenexecuted by the at least one first processors, further causes the visualoutput device to: based on a determination to present the one or moremedia content as a virtual reality presentation, and based on adetermination that the one or more media content is not already beingpresented as a virtual reality presentation, present, with the displaydevice and to the user, one or more second media content, the one ormore second media content comprising virtual reality content; and basedon a determination to present the one or more media content as a mixedreality presentation, and based on a determination that the one or moremedia content is not already being presented as a mixed realitypresentation, present, with the display device and to the user, one ormore third media content, the one or more third media content comprisingmixed reality content.

In another aspect, disclosed herein are methods, comprising:determining, with a computing system, how at least one of one or morefirst media content of a plurality of media content or one or morecaptured images of one or more physical objects should be presented to auser; based on a determination to present the one or more first mediacontent as a virtual reality presentation, and based on a determinationthat the one or more first media content is not already being presentedas a virtual reality presentation, controlling, with the computingsystem via a first API, an asset integration and control system to causethe asset integration and control system to modify the first mediacontent to generate one or more second media content, the one or moresecond media content comprising virtual reality content, and sending,with the computing system via a second API, the one or more second mediacontent to a first visual output device of a plurality of visual outputdevices that presents the one or more second media content to the user;based on a determination to present the one or more first media contentand the one or more captured images of the one or more physical objectsas an augmented reality presentation, and based on a determination thatthe one or more first media content and the one or more captured imagesof the one or more physical objects are not already being presented asan augmented reality presentation, controlling, with the computingsystem via the first API, the asset integration and control system tocause the asset integration and control system to generate one or morethird media content by modifying the one or more first media content asa virtual overlay in relation to the one or more captured images of theone or more physical objects, the one or more third media contentcomprising augmented reality content, and sending, with the computingsystem via the second API, the one or more third media content to thefirst visual output device that presents the one or more third mediacontent to the user; and based on a determination to present the one ormore first media content and the one or more captured images of the oneor more physical objects as a mixed reality presentation combiningelements of virtual reality and augmented reality, and based on adetermination that the one or more first media content and the one ormore captured images of the one or more physical objects are not alreadybeing presented as a mixed reality presentation, controlling, with thecomputing system via the first API, the asset integration and controlsystem to cause the asset integration and control system to generate oneor more fourth media content based on the one or more first mediacontent and the one or more captured images of the one or more physicalobjects, the one or more fourth media content comprising mixed realitycontent, and sending, with the computing system via the second API, theone or more fourth media content to the first visual output device thatpresents the one or more fourth media content to the user. In someembodiments, the method further comprises: tracking, with the computingsystem, a real-time status of each of the plurality of the media contentin each of at least one of the asset integration and control system viathe first API, the first visual output device via the second API, anasset production system via a third API, and a production managementsystem via a fourth API; distributing, with the computing system, atleast one fifth media content of the plurality of the media contentwhose status indicates that it is ready for integration from the assetproduction system to the asset integration and control system;distributing, with the computing system, at least one sixth mediacontent of the plurality of the media content whose status indicatesthat it is ready for distribution from the asset integration and controlsystem to the first visual output device; syncing, with the computingsystem, the at least one sixth media content with one or more secondvisual output devices of the plurality of visual output devices, the oneor more second visual output devices being separate from the firstvisual output device; and in response to receiving updates to the atleast one sixth media content, sending, with the computing system, theupdates to the at least one sixth media content to each of the firstvisual output device and each of the one or more second visual outputdevices. In some embodiments, the method further comprises: receiving,with the computing system, the one or more first media content from anasset production system via a third API; and receiving, with thecomputing system, the one or more captured images of the one or morephysical objects from an image capture device. In further embodiments,the method further comprises: determining, with the computing system,that the one or more first media content is ready for distribution tothe first visual output device; sending, with the computing system, oneor more first messages to a production management system via a fourthAPI, wherein the one or more first messages causes the productionmanagement system to change a status of the one or more first mediacontent to indicate that the one or more first media content is readyfor distribution to the first visual output device; and sending, withthe computing system, one or more second messages to the assetproduction system via the third API, wherein the one or more secondmessages causes the asset production system to send the one or morefirst media content to the computing system via the third API. In stillfurther embodiments, the method further comprises: determining, with thecomputing system, that one or more second media content is not ready fordistribution to the first visual output device; sending, with thecomputing system, one or more third messages to a production managementsystem via a fourth API, wherein the one or more third messages causesthe production management system to change a status of the one or moresecond media content to indicate that the one or more second mediacontent is not ready for distribution to the first visual output device;and sending, with the computing system, one or more fourth messages toan asset production system via the third API, wherein the one or morefourth messages causes the asset production system to send one or morefifth messages to one or more developers to continue work on the one ormore second media content. In some embodiments, the computing systemcomprises at least one of a server computer, a cloud computing system,or a computing system of the visual output device. In some embodiments,the plurality of visual output devices each comprises one of a VRheadset, an AR headset, a set of AR glasses, a tablet computer, a set ofbionic contact lenses, a smart phone adapted as part of a VR headset, ora smart phone adapted as part of an AR system.

In another aspect, disclosed herein are computing systems, comprising:at least one processor; and a non-transitory computer readable medium,the non-transitory computer readable medium having stored thereoncomputer software comprising a set of instructions that, when executedby the at least one processors, causes the computing system to:determine how at least one of one or more first media content of aplurality of media content or one or more captured images of one or morephysical objects should be presented to a user; based on a determinationto present the one or more first media content as a virtual realitypresentation, and based on a determination that the one or more firstmedia content is not already being presented as a virtual realitypresentation, control, via a first API, an asset integration and controlsystem to cause the asset integration and control system to modify thefirst media content to generate one or more second media content, theone or more second media content comprising virtual reality content, andsend, via a second API, the one or more second media content to a firstvisual output device of a plurality of visual output devices thatpresents the one or more second media content to the user; based on adetermination to present the one or more first media content and the oneor more captured images of the one or more physical objects as anaugmented reality presentation, and based on a determination that theone or more first media content and the one or more captured images ofthe one or more physical objects are not already being presented as anaugmented reality presentation, control, via the first API, the assetintegration and control system to cause the asset integration andcontrol system to generate one or more third media content by modifyingthe one or more first media content as a virtual overlay in relation tothe one or more captured images of the one or more physical objects, theone or more third media content comprising augmented reality content,and send, via the second API, the one or more third media content to thefirst visual output device that presents the one or more third mediacontent to the user; and based on a determination to present the one ormore first media content and the one or more captured images of the oneor more physical objects as a mixed reality presentation combiningelements of virtual reality and augmented reality, and based on adetermination that the one or more first media content and the one ormore captured images of the one or more physical objects are not alreadybeing presented as a mixed reality presentation, control, via the firstAPI, the asset integration and control system to cause the assetintegration and control system to generate one or more fourth mediacontent based on the one or more first media content and the one or morecaptured images of the one or more physical objects, the one or morefourth media content comprising mixed reality content, and send, via thesecond API, the one or more fourth media content to the first visualoutput device that presents the one or more fourth media content to theuser.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating a system for implementingcomputer-simulated reality interactions between users and physicalpublications, in accordance with various embodiments.

FIG. 2 is a flow diagram illustrating a method for implementingcomputer-simulated reality interactions between users and physicalpublications, in accordance with various embodiments.

FIG. 3 is a schematic diagram illustrating another system forimplementing computer-simulated reality interactions between users andphysical publications, in accordance with various embodiments.

FIG. 4 is a flow diagram illustrating another method for implementingcomputer-simulated reality interactions between users and physicalpublications, in accordance with various embodiments.

FIGS. 5A-5D are flow diagrams illustrating a method for implementingcomputer-simulated reality interactions between users and physicalobjects, in accordance with various embodiments.

FIG. 6 is a schematic diagram illustrating a system for implementing acomputer-simulated reality platform that enables shifting from one ofvirtual reality, augmented reality, or mixed reality to another ofvirtual reality, augmented reality, or mixed reality, in accordance withvarious embodiments.

FIGS. 7A-7F are flow diagrams illustrating a method for implementing acomputer-simulated reality platform that enables shifting from one ofvirtual reality, augmented reality, or mixed reality to another ofvirtual reality, augmented reality, or mixed reality, in accordance withvarious embodiments.

FIG. 8 is a block diagram illustrating an exemplary computer or systemhardware architecture, in accordance with various embodiments.

FIG. 9 is a block diagram illustrating a networked system of computers,which can be used in accordance with various embodiments.

FIGS. 10-25 illustrate an exemplary e-book.

FIGS. 26-30 illustrate an exemplary AR experience.

FIG. 31 illustrates a first exemplary VR experience.

FIGS. 32-36 illustrate a second exemplary VR experience.

DETAILED DESCRIPTION OF THE INVENTION Certain Definitions

Unless otherwise defined, all technical terms used herein have the samemeaning as commonly understood by one of ordinary skill in the art towhich this invention belongs. As used in this specification and theappended claims, the singular forms “a,” “an,” and “the” include pluralreferences unless the context clearly dictates otherwise. Any referenceto “or” herein is intended to encompass “and/or” unless otherwisestated.

A used herein, “trigger” includes an image, frame marker (e.g., QRcode), text, physical three-dimensional object, and the like, used toidentify and/or activate AR, VR, or MxR media content.

As used herein, “non-visual trigger” includes a Near FieldCommunications (NFC) tag, Radio Frequency Identification Data (RFID)Mode 2 or 6 tag, Location Based Service (LBS) active on a smartphone,and the like, used to identify and/or activate AR, VR, or MxR mediacontent.

Various embodiments provide tools and techniques for implementingcomputer-simulated reality interactions, and, in particular embodiments,to methods, systems, and computer software for implementingcomputer-simulated reality interactions between users and physicalpublications.

Computer-Simulated Reality User Experience (“UX”)

In various embodiments, a first camera might capture one or more imagesof at least a portion of a printed publication, and a computing systemmight analyze the captured one or more images to determine whether thecaptured one or more images contain one or more triggers. Based on adetermination that the captured one or more images contain at least onetrigger of the one or more triggers, the computing system might identifyone or more media content of a plurality of media content that areassociated with the at least one trigger. The computing system mightdetermine a field of view of a user (e.g., by capturing an eye(s) of theuser using a camera in relation to what the user is looking at), thefield of view including at least a portion of the printed publication.The computing system might subsequently modify an orientation of the oneor more media content to align with the field of view of the user, ifnecessary (i.e., if the computing system determines that the orientationof the one or more media content are not already aligned with the fieldof view of the user, or the like). The one or more media content (insome cases, whose orientation has been modified to align with the fieldof view of the user) may then be presented with a visual output deviceand to the user, such that the one or more media content is presented asa virtual overlay in relation to at least a portion of the printedpublication.

Alternatively, rather than using visual triggers, non-visual triggersmay be used. In such cases, wireless communications may be establishedbetween a visual output device and at least one trigger of one or morenon-visual triggers associated with a printed publication. In responseto establishing communications between the visual output device and theat least one trigger, the computing system might identify one or moremedia content of a plurality of media content that are associated withthe at least one trigger. The computing system might determine a fieldof view of a user (e.g., by capturing an eye(s) of the user using acamera in relation to what the user is looking at), the field of viewincluding at least a portion of the printed publication. The computingsystem might subsequently modify an orientation of the one or more mediacontent to align with the field of view of the user, if necessary (i.e.,if the computing system determines that the orientation of the one ormore media content are not already aligned with the field of view of theuser, or the like). The one or more media content (in some cases, whoseorientation has been modified to align with the field of view of theuser) may then be presented with a visual output device and to the user,such that the one or more media content is presented as a virtualoverlay in relation to the at least a portion of the printedpublication.

In some embodiments, the printed publication might include, withoutlimitation, one of a fiction novel, a non-fiction novel, a children'sbook, a newspaper, a magazine, a comic book, a comic strip, a graphicnovel, an encyclopedia, a dictionary, a map, a calendar, a greetingcard, a bookmark, a venue or event ticket stub, or a poster, and/or thelike. According to some embodiments, the at least one visual triggermight each include, without limitation, one of a quick response (“QR”)code provided on the at least a portion of the publication, a barcodeprovided on the at least a portion of the publication, a graphical codeprovided on a sticker affixed to the at least a portion of thepublication, a graphic printed on the at least a portion of thepublication, a keyword or key code printed on the at least a portion ofthe publication, a cover image of the printed publication, a titleprinted on a cover of the printed publication, author name printed onthe cover of the printed publication, a stuffed character associatedwith the printed publication that is positioned in proximity to theprinted publication, a 3D-printed character associated with the printedpublication that is positioned in proximity to the printed publication,or a 3D-printed object associated with the printed publication that ispositioned in proximity to the printed publication, and/or the like. Theat least one non-visual trigger, in some cases, might each include, butis not limited to, one of a radio frequency identification (“RFED”) tag,a Bluetooth low energy (“BLE”) tag, a near field communication (“NFC”)tag, or a real-time locating systems (“RTLS”) tag, and/or the like.

In some instances, the one or more media content comprises at least oneof a three-dimensional (“3D”) still graphic, a 3D animated graphic, or a3D interactive animated graphic, and/or the like, each of which mightinclude, without limitation, one of a graphic representation of one ormore characters associated with the printed publication, a non-fictionalgeographical location associated with the printed publication, afictional geographical location associated with the printed publication,a room associated with the printed publication, a building associatedwith the printed publication, a scene described in the printedpublication, a scene inspired by the scene described in the printedpublication, one or more items described in the printed publication, orone or more items inspired by the one or more items described in theprinted publication, and/or the like.

In some embodiments, the visual output device might include, withoutlimitation, one of a VR headset, an AR headset, a set of AR glasses, atablet computer, a set of bionic contact lenses, a smart phone adaptedas part of a VR headset, or a smart phone adapted as part of an ARsystem, and/or the like. According to some embodiments, presenting theone or more media content might comprise presenting, with the visualoutput device and to the user, the one or more media content whoseorientation has been modified to align with the field of view of theuser, such that the one or more media content is at least one ofpresented as a virtual overlay above a cover of the printed publication,presented as a virtual overlay on a cover of the printed publication,presented as a virtual overlay beside the printed publication, presentedas a virtual overlay above a page of the printed publication, presentedas a virtual overlay on a page of the printed publication, presented asa virtual overlay in front of a surface of the printed publication,presented as a virtual overlay in front of a binding of the printedpublication, presented as a virtual overlay above a two-dimensional(“2D”) graphic printed on a portion of the printed publication, orpresented as a virtual overlay on a 2D graphic printed on a portion ofthe printed publication, and/or the like.

In some embodiments, the computing system described above might includecomputer vision (in some cases, with object recognition). Here,“computer vision” might refer to image capture technology thatrecognizes objects and triggers, and that initiates AR presentations ofcontent in response to detecting the triggers (which could bevisual-based triggers, non-visual-based triggers, any combination ofvisual and non-visual triggers, and/or the like). In some instances,computer vision technology might further comprise using AR as a bridgeinto VR, and vice versa, where, in some cases, the computer visionsystem might dynamically or selectively shift between AR and VRdepending on factors that would enhance the interactive experience forthe user or reader (and which might be based on detecting andidentifying user responses to certain stimuli in the AR and/or VRpresentations). Herein, “augmented reality” or “AR” might refer tovirtual overlay of simulated constructs either over actual views ofactual objects and settings or over images of actual objects andsettings, while “virtual reality” or “VR” might refer to an enclosedsensory environment where everything that is observed by the user issimulated, and “mixed reality” or “MxR” might refer to a combination ofAR and VR (e.g., a VR presentation in which AR elements are simulatedare embedded in the presentation, or the like). These and otherimplementations of computer vision are described in detail below withrespect to the various non-limiting embodiments.

The virtual overlay of characters, story elements, settings, scenes,and/or the like that are associated with the printed publicationprovides the user or reader with a virtual interaction (aside from theuser's or reader's imagination) with the characters, the settings,and/or the story associated with the publication. In some cases, suchinteractions include interactive games that the user plays with thevirtual characters from the story of the publication. Some embodimentswould also allow the user or reader to experience, via the AR, VR,and/or mixed reality (i.e., combination of AR and VR), the story fromthe point of view of a protagonist or an antagonist in the story,thereby enhancing the user's or reader's sense of cognitive andeffective or affected empathy for the characters or the story in thepublication. Further, the dynamic or selected shifts between AR and VRprovides the user or reader with a more immersive interactive experiencewith the story and/or the characters in the publication.

Computer-Simulated Reality Platform

The computer-simulated reality platform is a scalable and reusableframework and end user application for a MxR media content application,with an AR and 2D front-end launching functionality into immersive VR.The computer-simulated reality platform provides an end-to-endproduction process and media content application framework that enablesa streamlined and cost-effective production process, which is trackedand managed from first design of the media content presentation throughpublication and analytics. In some cases, the framework, which is acompletely custom client/server technology, provides for creation,distribution, and management of every media content asset or application(herein also referred to as “assets,” “media content,” “application,”etc.) that is released into the proprietary application framework. Theplatform includes, without limitation, a graphic representation of auser interface (“un and/or user experience (“UX”), architectural andserver-based functionality (e.g., for hosted content), technicalrequirements and a content management solution (“CMS”) for the hostingand support of the development and live versions of the application. Theplatform is intended to be universally adaptable to any MxR-basedplatforms as well as development projects including, but not limited to,repurposing of assets for episodic television.

Pre-Production Planning and Scheduling:

The traditional method of tracking production of assets (e.g.,Microsoft° EXCEL®, Google® Docs, e-mail applications, messagingapplications, etc.) have the disadvantage of being disconnected andtypically one-way in terms of communication. The platform creates anall-in-one tracking and management solution that takes the place ofthese traditional means of tracking production of assets, whileproviding direct connection and launching capabilities with assetproduction systems and software, including, but not limited to,Autodesk®^(Maya®), Autodesk® Mudbox°, Adobe° Photoshop®, and/or thelike. Using the software as a service (“SaaS”) model, custom productionprocesses have been built-out, in some cases using pre-productionplanning and scheduling tools (e.g., Autodesk Shotgun, or the like).Custom interfaces for every stage of production have been created toaccurately track design, development, artists' tasks, media contentstructure and tasking, animations, audio, as well as the bug managementsystem for tracking bugs in release versions and beta version of theapplications, and these are tracked in a redundantly backed-up, 128-bit(or better) encrypted online database.

Using proprietary tracking and production templates, the platform canmanage both internal and external artists, developers, writers, etc.(herein collectively referred to as “developers”) to scale productionson books exponentially, all while maintaining a tight level of qualitycontrol, with a minimal amount of production overhead.

In some embodiments, an internal toolset of the platform (e.g., QuantumBridge™ or the like), which might run on one or more of desktopcomputers, laptop computers, mobile devices, tablet platforms, and/orthe like, might connect all of the internal software and developmenttools via custom workflows (e.g., Quantum Transport™, and/or the like),which might be written in languages such as Python, JavaScript, and C#,and/or the like. The custom workflow application programming interface(“API”) might provide each disparate section or component of thedevelopment pipeline with ability to communicate with each other,sending messages back and forth when digital assets change their statusfrom pre-production phase to asset integration or to asset distribution,when the asset has been flagged as ready for integration or ready fordistribution, etc.

According to some embodiments, a production tracking, planning, andtasking system might be used to develop custom workflows that fitparticular pre-production and production styles of the developers, whileenabling improvement and modification as production progresses. Theproduction tracking, planning, and tasking system can track artists'time, track deliveries on every asset required for creation of newdigital stories, create dependent tasks (where one task starts whenanother is finished, where one task starts when another has progressedto a certain stage, etc.), and track versions of all of the above. Theproduction tracking, planning, and tasking system can also seeddevelopment, testing, and production database on the server or computingsystem, for delivery of assets. Using built-in APIs and/or the platformtoolset, as assets are flagged for approval, the resulting files can besynchronized or synced through custom scripts (e.g., Quantum Triggerscripts, or the like) that push and convert the data on the productionmanagement system into asset bundles for the asset production system(e.g., Unity 3D, etc.) for viewing and reviewing in the framework in theasset production system.

From planning out timelines, to costing per asset (e.g., models, sounds,animations, etc.), the platform will be able to accurately schedule andmanage the time/cost using scheduling tools (e.g., in the productionmanagement system, or the like) as well as custom tools in the platformtoolset. With these, the platform can track and predict new assets asthe process progresses from concept to completion, through todistribution and delivery. This is important for each asset that ispublished, where timelines in which they are created may constantlyrequire revision and where the workforce or developers required tocomplete each stage may need to be changed.

Asset Production:

Typically, production in 3D or 2D takes on a more manual assignment anditeration process that creates many single points of failure frombuilding the assets to seeing the latest content updated inside theapplication. In some cases, some production methodologies are notscalable for multistory solutions that are being deployed.

The platform, using tools (e.g., the production management system tools,the platform toolset, industry standard software for asset creation,and/or the like) with tasking, review, and approval functionalities, canbuild, track, and deploy assets in a much more streamlined fashion.Every asset created in the production process can be connected viameta-data, scripts, and/or the like, via codes created in the platformtoolset, for accurate tracking, as well as easier digital distributionand updating to end users' devices. The platform can task artists andother developers directly, and, with custom scripts, can push approvedversions of assets directly from the production suite of tools into thedevelopment testing areas for review in VR-based applications, AR-basedapplications, or MxR-based applications, or a combination of theseapplications, and the like.

Asset Integration and Control:

Custom tools in the platform can tie in data from the earliest stages ofpre-production. Using a data-based approach allows for tracking everyasset in the asset builds that are tested and that subsequently gopublic. This not only helps with quality control, but also helps withcost overruns, time delays, and content bugs that are highly typical inmore manual-based development workflows. Using a multi-platform assetproduction system, very customizable appearances and assets (in both 2Dand 3D implementations) can be made. The multi-platform functionalityallows for development within one codebase that with minor tweaksenables pushing the framework to any one of a plurality of differentpresentation platforms. Computer vision technology and the correspondingsystem can be used to allow the asset integration system to recognizeand track real-world objects and images with a camera on a mobile devicefor an augmented reality-based experience (or a mixed reality-basedexperience). The images can be hosted in the application (or app), ordynamically served from a computer vision development website.

Asset Integration and Control:

A server (e.g., Parse Server, etc.) might allow for storing anddelivering large amounts of data digitally to the framework, and mighttie directly into the asset integration systems, as well as otherdevelopment platforms. The server might be highly scalable horizontallyand vertically and might run seamlessly on currently available webservice solutions (e.g., Amazon® Web Services, or the like). Otherbenefits might include, without limitation, user login and accountcreation, custom database schemas, oAuth2 connections for easier parentaccounts, storing of Children's Online Privacy Protection Act (COPPA)compliancy information, push notification framework and server, and/orthe like.

The asset integration system might tie directly in the data that ishosted in the server(s), which in turn might run on the web servicesolutions. The server data can be synced between the productionmanagement system and the server, with hooks into the assets created inthe asset production system and the asset integration system, etc. Thisallows for tracking assets and versions from pre-production all the waythrough to delivery to release of the platform. The server can trackdevices, assets, and users, and can create custom analytics that willdrive better user retention and monetization of the framework acrossevery platform on which the asset is available.

Delivery Platforms:

Using the asset integration system, and delivery of the asset or mediacontent dynamically through the server(s), multiple platforms can bequickly supported and at a much lower cost than manual delivery methods.Various delivery platforms can be used, including, but not limited to,Google Cardboard, Samsung Gear VR°, Google Daydream™, MicrosoftHoloLens™, Oculus, HTC Vive™, Sony PlayStation VR°, and/or the like.

Syncing:

Using proprietary tools, both in the server and the front-end client,the server controls all aspects of the user experience. Syncing with theproduction management system, the server maintains the latest,up-to-date information on assets, updates, users, devices, and the like,and runs up-to-the-minute analytics on anything and everything relatedto the applications, assets, etc., both for testing as well as forrelease.

Users, COPPA compliancy tracking, application flow and structure, etc.can be managed through custom interfaces into the data through theservers. The servers utilize various tools (e.g., MongoDB°, Node.js,ReST, custom APIs, etc.) that communicate with client devices, which maybe located anywhere in the world. They rely on a stateless,client-server, cacheable communications protocols, with the ability toscale to millions of users dynamically.

Utilizing the multiversion concurrency control (“MVCC”) approach todatabase management allows the platform to update data in clientapplications within minutes. Changing story points or flow, removing oradding assets, and the like are straightforward and immediatelyavailable to all of the client devices.

Users, installations, geo-locations, time spent in the user experiences,and/or the like, down to the granular level of time spent in a specificlocation (e.g., virtual location, etc.) in an asset application can betracked and fed through the servers for custom analytics that can beused to track retention, user funnels for events, and a myriad of customreports that allow for continual refinement and honing of the processfor offering the user experience, using real customer data.

The following detailed description illustrates a few exemplaryembodiments in further detail to enable one of skill in the art topractice such embodiments. The described examples are provided forillustrative purposes and are not intended to limit the scope of theinvention.

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the described embodiments. It will be apparent to oneskilled in the art, however, that other embodiments of the presentinvention may be practiced without some of these specific details. Inother instances, certain structures and devices are shown in blockdiagram form. Several embodiments are described herein, and whilevarious features are ascribed to different embodiments, it should beappreciated that the features described with respect to one embodimentmay be incorporated with other embodiments as well. By the same token,however, no single feature or features of any described embodimentshould be considered essential to every embodiment of the invention, asother embodiments of the invention may omit such features.

The tools provided by various embodiments include, without limitation,methods, systems, and/or software products. Merely by way of example, amethod might comprise one or more procedures, any or all of which areexecuted by a computer system. Correspondingly, an embodiment mightprovide a computer system configured with instructions to perform one ormore procedures in accordance with methods provided by various otherembodiments. Similarly, a computer program might comprise a set ofinstructions that are executable by a computer system (and/or aprocessor therein) to perform such operations. In many cases, suchsoftware programs are encoded on physical, tangible, and/ornon-transitory computer readable media (such as, to name but a fewexamples, optical media, magnetic media, and/or the like).

Various embodiments described herein, while embodying (in some cases)software products, computer-performed methods, and/or computer systems,represent tangible, concrete improvements to existing technologicalareas, including, without limitation, augmented reality technology,virtual reality technology, mixed reality technology, interactive gamingtechnology, and/or the like. In other aspects, certain embodiments, canimprove the functioning of a computer or network system itself (e.g.,computing devices or systems, user devices, or the like for performingthe functionalities described below, etc.), for example, by determining,with a computer system, a field of view of a user, the field of viewincluding at least a portion of the printed publication, modifying, withthe computer system, an orientation of the one or more media content toalign with the field of view of the user, and presenting, with a visualoutput device and to the user, the one or more media content whoseorientation has been modified to align with the field of view of theuser, such that the one or more media content is presented as a virtualoverlay in relation to the at least a portion of the printedpublication, thereby improving the functioning of an augmented orvirtual reality interface for a user by generating virtual projectionsof media content that seem more realistic, by shifting between two ofaugmented reality, mixed reality, and virtual reality in a seamlessmanner (which requires simulation and modification of media content tomaintain proper perspectives particularly as the user is moving aboutand changing his or her field of view), and presenting media content inAR, mixed reality, or VR based on the shifting, and/or the like. Inparticular, to the extent any abstract concepts are present in thevarious embodiments, those concepts can be implemented as describedherein by devices, software, systems, and methods that involve specificnovel functionality (e.g., steps or operations), such as modifying, withthe computer system, an orientation of the one or more media content toalign with the field of view of the user, and presenting, with a visualoutput device and to the user, the one or more media content whoseorientation has been modified to align with the field of view of theuser, such that the one or more media content is presented as a virtualoverlay in relation to the at least a portion of the printedpublication, or enabling shifting between two of augmented reality,mixed reality, and virtual reality in a seamless manner (which requiressimulation and modification of media content to maintain properperspectives particularly as the user is moving about and changing hisor her field of view), and presenting media content in AR, mixedreality, or VR based on the shifting, and/or the like, to name a fewexamples, that extend beyond mere conventional computer processingoperations. These functionalities can produce tangible results outsideof the implementing computer system, including, merely by way ofexample, ability to generate 3D virtual projections of media content inrelation to physical publications that seem realistic to a user, abilityto seamlessly transition between two of augmented reality, mixedreality, and virtual reality thereby providing the user with animmersive interactive environment, and/or the like, which may beobserved or measured by customers and/or service providers.

In an aspect, a method might comprise capturing, with a first camera,one or more images of at least a portion of a printed publication;analyzing, with a computing system, the captured one or more images ofthe at least a portion of the printed publication to determine whetherthe captured one or more images contain one or more triggers; and, basedon a determination that the captured one or more images contain at leastone trigger of the one or more triggers, identifying, with the computingsystem, one or more media content of a plurality of media content thatare associated with the at least one trigger. The method might alsocomprise determining, with the computer system, a field of view of auser, the field of view including at least a portion of the printedpublication and modifying, with the computer system, an orientation ofthe one or more media content to align with the field of view of theuser. The method might further comprise presenting, with a visual outputdevice and to the user, the one or more media content whose orientationhas been modified to align with the field of view of the user, such thatthe one or more media content is presented as a virtual overlay inrelation to the at least a portion of the printed publication.

In some embodiments, the visual output device might comprise one of a VRheadset, an AR headset, a set of AR glasses, a tablet computer, a set ofbionic contact lenses, a smart phone adapted as part of a VR headset, ora smart phone adapted as part of an AR system, and/or the like. In somecases, the visual output device might comprise the first camera.Alternatively, the first camera might be external to the visual outputdevice, while being communicatively coupled to the visual output device.

According to some embodiments, the visual output device might comprise asecond camera that captures one or more images of eyes of the user asthe first camera captures the one or more images of the at least aportion of the printed publication, and determining, with the computersystem, a field of view of a user might comprise determining, with thecomputer system, a field of view of a user based on an analysis of thecaptured one or more images of the eyes of the user.

In some instances, the printed publication might comprise one of afiction novel, a nonfiction novel, a children's book, a newspaper, amagazine, a comic book, a comic strip, a graphic novel, an encyclopedia,a dictionary, a map, a calendar, a greeting card, a bookmark, a venue orevent ticket stub, or a poster, and/or the like. The at least onetrigger, according to some embodiments, might each comprise one of a QRcode provided on the at least a portion of the publication, a barcodeprovided on the at least a portion of the publication, a graphical codeprovided on a sticker affixed to the at least a portion of thepublication, a graphic printed on the at least a portion of thepublication, a keyword or key code printed on the at least a portion ofthe publication, a cover image of the printed publication, a titleprinted on a cover of the printed publication, author name printed onthe cover of the printed publication, a stuffed character associatedwith the printed publication that is positioned in proximity to theprinted publication, a 3D-printed character associated with the printedpublication that is positioned in proximity to the printed publication,or a 3D-printed object associated with the printed publication that ispositioned in proximity to the printed publication, and/or the like.

In some cases, the one or more media content might comprise at least oneof a 3D still graphic, a 3D animated graphic, or a 3D interactiveanimated graphic, and/or the like. Each of the 3D still graphic, the 3Danimated graphic, or the 3D interactive animated graphic might compriseat least one of a graphic representation of one or more charactersassociated with the printed publication, a non-fictional geographicallocation associated with the printed publication, a fictionalgeographical location associated with the printed publication, a roomassociated with the printed publication, a building associated with theprinted publication, a scene described in the printed publication, ascene inspired by the scene described in the printed publication, one ormore items described in the printed publication, or one or more itemsinspired by the one or more items described in the printed publication,and/or the like.

In some embodiments, presenting the one or more media content mightcomprise presenting, with the visual output device and to the user, theone or more media content whose orientation has been modified to alignwith the field of view of the user, such that the one or more mediacontent is at least one of presented as a virtual overlay above a coverof the printed publication, presented as a virtual overlay on a cover ofthe printed publication, presented as a virtual overlay beside theprinted publication, presented as a virtual overlay above a page of theprinted publication, presented as a virtual overlay on a page of theprinted publication, presented as a virtual overlay in front of asurface of the printed publication, presented as a virtual overlay infront of a binding of the printed publication, presented as a virtualoverlay above a 2D graphic printed on a portion of the printedpublication, or presented as a virtual overlay on a 2D graphic printedon a portion of the printed publication, and/or the like. In some cases,the computing system might comprise at least one of a server computer, acloud computing system, or a computing system of the visual outputdevice, and/or the like.

In another aspect, an apparatus might comprise a first camera, a displaydevice, at least one processor, and a non-transitory computer readablemedium. The non-transitory computer readable medium might have storedthereon computer software comprising a set of instructions that, whenexecuted by the at least one processors, causes the apparatus to:capture, with the first camera, one or more images of at least a portionof a printed publication; analyze the captured one or more images of theat least a portion of the printed publication to determine whether thecaptured one or more images contain one or more triggers; based on adetermination that the captured one or more images contain at least onetrigger, identify one or more media content of a plurality of mediacontent that are associated with the at least one trigger; determine afield of view of a user, the field of view including at least a portionof the printed publication; modify an orientation of the one or moremedia content to align with the field of view of the user; and present,with the display device and to the user, the one or more media contentwhose orientation has been modified to align with the field of view ofthe user, such that the one or more media content is presented as avirtual overlay in relation to the at least a portion of the printedpublication.

In some embodiments, the apparatus might comprise one of a VR headset,an AR headset, a set of AR glasses, a tablet computer, a set of bioniccontact lenses, a smart phone adapted as part of a VR headset, or asmart phone adapted as part of an AR system, and/or the like.

In some cases, the apparatus might further comprise a second camera. Theset of instructions, when executed by the one or more processors, mightfurther cause the apparatus to: capture, with the second camera, one ormore images of eyes of the user as the first camera captures the one ormore images of the at least a portion of the printed publication.Determining a field of view of a user might comprise determining a fieldof view of a user based on an analysis of the captured one or moreimages of the eyes of the user.

According to some embodiments, the printed publication might compriseone of a fiction novel, a non-fiction novel, a children's book, anewspaper, a magazine, a comic book, a comic strip, a graphic novel, anencyclopedia, a dictionary, a map, a calendar, a greeting card, abookmark, a venue or event ticket stub, or a poster, and/or the like. Insome cases, the at least one trigger might each comprise one of a QRcode provided on the at least a portion of the publication, a barcodeprovided on the at least a portion of the publication, a graphical codeprovided on a sticker affixed to the at least a portion of thepublication, a graphic printed on the at least a portion of thepublication, a keyword or key code printed on the at least a portion ofthe publication, a cover image of the printed publication, a titleprinted on a cover of the printed publication, author name printed onthe cover of the printed publication, a stuffed character associatedwith the printed publication that is positioned in proximity to theprinted publication, a 3D-printed character associated with the printedpublication that is positioned in proximity to the printed publication,or a 3D-printed object associated with the printed publication that ispositioned in proximity to the printed publication, and/or the like.

Merely by way of example, the one or more media content might compriseat least one of a 3D still graphic, a 3D animated graphic, or a 3Dinteractive animated graphic, and/or the like. Each of the 3D stillgraphic, the 3D animated graphic, or the 3D interactive animated graphicmight comprise at least one of a graphic representation of one or morecharacters associated with the printed publication, a non-fictionalgeographical location associated with the printed publication, afictional geographical location associated with the printed publication,a room associated with the printed publication, a building associatedwith the printed publication, a scene described in the printedpublication, a scene inspired by the scene described in the printedpublication, one or more items described in the printed publication, orone or more items inspired by the one or more items described in theprinted publication, and/or the like.

In some embodiments, presenting the one or more media content mightcomprise presenting, with the display device and to the user, the one ormore media content whose orientation has been modified to align with thefield of view of the user, such that the one or more media content is atleast one of presented as a virtual overlay above a cover of the printedpublication, presented as a virtual overlay on a cover of the printedpublication, presented as a virtual overlay beside the printedpublication, presented as a virtual overlay above a page of the printedpublication, presented as a virtual overlay on a page of the printedpublication, presented as a virtual overlay in front of a surface of theprinted publication, presented as a virtual overlay in front of abinding of the printed publication, presented as a virtual overlay abovea 2D graphic printed on a portion of the printed publication, orpresented as a virtual overlay on a 2D graphic printed on a portion ofthe printed publication, and/or the like.

In yet another aspect, a system might comprise an image capture device,a computing system, and a visual output device. The image capture devicemight comprise a first camera, at least one first processor, and a firstnon-transitory computer readable medium. The first non-transitorycomputer readable medium might have stored thereon computer softwarecomprising a first set of instructions that, when executed by the atleast one first processors, causes the image capture device to: capture,with the first camera, one or more images of at least a portion of aprinted publication and send the captured one or more images to thecomputing system.

The computing system might comprise at least one second processor and asecond non-transitory computer readable medium. The secondnon-transitory computer readable medium might have stored thereoncomputer software comprising a second set of instructions that, whenexecuted by the at least one second processors, causes the computingsystem to: receive the captured one or more images from the imagecapture device; analyze the captured one or more images of the at leasta portion of the printed publication to determine whether the capturedone or more images contain one or more triggers; based on adetermination that the captured one or more images contain at least onetrigger, identify one or more media content of a plurality of mediacontent that are associated with the at least one trigger; determine afield of view of a user, the field of view including at least a portionof the printed publication; modify an orientation of the one or moremedia content to align with the field of view of the user; and send theone or more media content whose orientation has been modified to alignwith the field of view of the user to the visual output device.

The visual output device might comprise a display device, at least onethird processor, and a third non-transitory computer readable medium.The third non-transitory computer readable medium might have storedthereon computer software comprising a third set of instructions that,when executed by the at least one third processors, causes the visualoutput device to: receive the one or more media content whoseorientation has been modified to align with the field of view of theuser from the computing system; and present, with the display device andto the user, the one or more media content whose orientation has beenmodified to align with the field of view of the user, such that the oneor more media content is presented as a virtual overlay in relation tothe at least a portion of the printed publication.

In some embodiments, the visual output device might comprise one of a VRheadset, an AR headset, a set of AR glasses, a tablet computer, a set ofbionic contact lenses, a smart phone adapted as part of a VR headset, ora smart phone adapted as part of an AR system, and/or the like. In somecases, the visual output device might comprise the image capture device.

According to some embodiments, the visual output device might comprise asecond camera that captures one or more images of eyes of the user asthe first camera captures the one or more images of the at least aportion of the printed publication, and determining, with the computersystem, a field of view of a user might comprise determining, with thecomputer system, a field of view of a user based on an analysis of thecaptured one or more images of the eyes of the user.

In some instances, the printed publication might comprise one of afiction novel, a nonfiction novel, a children's book, a newspaper, amagazine, a comic book, a comic strip, a graphic novel, an encyclopedia,a dictionary, a map, a calendar, a greeting card, a bookmark, a venue orevent ticket stub, or a poster, and/or the like. The at least onetrigger, according to some embodiments, might each comprise one of a QRcode provided on the at least a portion of the publication, a barcodeprovided on the at least a portion of the publication, a graphical codeprovided on a sticker affixed to the at least a portion of thepublication, a graphic printed on the at least a portion of thepublication, a keyword or key code printed on the at least a portion ofthe publication, a cover image of the printed publication, a titleprinted on a cover of the printed publication, author name printed onthe cover of the printed publication, a stuffed character associatedwith the printed publication that is positioned in proximity to theprinted publication, a 3D-printed character associated with the printedpublication that is positioned in proximity to the printed publication,or a 3D-printed object associated with the printed publication that ispositioned in proximity to the printed publication, and/or the like.

In some cases, the one or more media content might comprise at least oneof a 3D still graphic, a 3D animated graphic, or a 3D interactiveanimated graphic, and/or the like. Each of the 3D still graphic, the 3Danimated graphic, or the 3D interactive animated graphic might compriseat least one of a graphic representation of one or more charactersassociated with the printed publication, a non-fictional geographicallocation associated with the printed publication, a fictionalgeographical location associated with the printed publication, a roomassociated with the printed publication, a building associated with theprinted publication, a scene described in the printed publication, ascene inspired by the scene described in the printed publication, one ormore items described in the printed publication, or one or more itemsinspired by the one or more items described in the printed publication,and/or the like.

In some embodiments, presenting the one or more media content mightcomprise presenting, with the visual output device and to the user, theone or more media content whose orientation has been modified to alignwith the field of view of the user, such that the one or more mediacontent is at least one of presented as a virtual overlay above a coverof the printed publication, presented as a virtual overlay on a cover ofthe printed publication, presented as a virtual overlay beside theprinted publication, presented as a virtual overlay above a page of theprinted publication, presented as a virtual overlay on a page of theprinted publication, presented as a virtual overlay in front of asurface of the printed publication, presented as a virtual overlay infront of a binding of the printed publication, presented as a virtualoverlay above a 2D graphic printed on a portion of the printedpublication, or presented as a virtual overlay on a 2D graphic printedon a portion of the printed publication, and/or the like. In some cases,the computing system might comprise at least one of a server computer, acloud computing system, or a computing system of the visual outputdevice, and/or the like.

In still another aspect, a method might comprise establishingcommunications between a visual output device and at least one triggerof one or more non-visual triggers associated with a printed publicationand, in response to establishing communications between the visualoutput device and the at least one trigger, identifying, with thecomputing system, one or more media content of a plurality of mediacontent that are associated with the at least one trigger. The methodmight also comprise determining, with the computer system, a field ofview of a user, the field of view including at least a portion of theprinted publication and modifying, with the computer system, anorientation of the one or more media content to align with the field ofview of the user. The method might further comprise presenting, with thevisual output device and to the user, the one or more media contentwhose orientation has been modified to align with the field of view ofthe user, such that the one or more media content is presented as avirtual overlay in relation to the at least a portion of the printedpublication.

According to some embodiments, establishing communications between thevisual output device and the at least one trigger might comprise:determining, with the computing system, whether the visual output deviceis within communications range of the one or more non-visual triggersassociated with the printed publication; and, based on a determinationthat the visual output device is within communications range of the atleast one trigger of the one or more non-visual triggers, establishingcommunications between the visual output device and the at least onetrigger.

In some embodiments, the visual output device might comprise one of a VRheadset, an AR headset, a set of AR glasses, a tablet computer, a set ofbionic contact lenses, a smart phone adapted as part of a VR headset, ora smart phone adapted as part of an AR system, and/or the like. In somecases, the visual output device might comprise a camera that capturesone or more images of eyes of the user as the user is looking toward theat least a portion of the printed publication, and determining, with thecomputer system, a field of view of a user comprises determining, withthe computer system, a field of view of a user based on an analysis ofthe captured one or more images of the eyes of the user.

Merely by way of example, in some cases, the printed publication mightcomprise one of a fiction novel, a non-fiction novel, a children's book,a newspaper, a magazine, a comic book, a comic strip, a graphic novel,an encyclopedia, a dictionary, a map, a calendar, a greeting card, abookmark, a venue or event ticket stub, or a poster, and/or the like. Insome instances, the at least one trigger might each comprise one of aRFID tag, a BLE tag, a NFC tag, or a RTLS tag, and/or the like.

In some cases, the one or more media content might comprise at least oneof a 3D still graphic, a 3D animated graphic, or a 3D interactiveanimated graphic, and/or the like. Each of the 3D still graphic, the 3Danimated graphic, or the 3D interactive animated graphic might compriseat least one of a graphic representation of one or more charactersassociated with the printed publication, a non-fictional geographicallocation associated with the printed publication, a fictionalgeographical location associated with the printed publication, a roomassociated with the printed publication, a building associated with theprinted publication, a scene described in the printed publication, ascene inspired by the scene described in the printed publication, one ormore items described in the printed publication, or one or more itemsinspired by the one or more items described in the printed publication,and/or the like.

In some embodiments, presenting the one or more media content mightcomprise presenting, with the visual output device and to the user, theone or more media content whose orientation has been modified to alignwith the field of view of the user, such that the one or more mediacontent is at least one of presented as a virtual overlay above a coverof the printed publication, presented as a virtual overlay on a cover ofthe printed publication, presented as a virtual overlay beside theprinted publication, presented as a virtual overlay above a page of theprinted publication, presented as a virtual overlay on a page of theprinted publication, presented as a virtual overlay in front of asurface of the printed publication, presented as a virtual overlay infront of a binding of the printed publication, presented as a virtualoverlay above a 2D graphic printed on a portion of the printedpublication, or presented as a virtual overlay on a 2D graphic printedon a portion of the printed publication, and/or the like. In some cases,the computing system might comprise at least one of a server computer, acloud computing system, or a computing system of the visual outputdevice, and/or the like.

In another aspect, an apparatus might comprise a wireless communicationsdevice, a display device, at least one processor, and a non-transitorycomputer readable medium. The non-transitory computer readable mediummight have stored thereon computer software comprising a set ofinstructions that, when executed by the at least one processors, causesthe apparatus to: establish, with the wireless communications device,communications between a visual output device and at least one triggerof one or more non-visual triggers associated with a printedpublication; in response to establishing communications between thevisual output device and the at least one trigger, identify one or moremedia content of a plurality of media content that are associated withthe at least one trigger; determine a field of view of a user, the fieldof view including at least a portion of the printed publication; modifyan orientation of the one or more media content to align with the fieldof view of the user; and present, with the display device and to theuser, the one or more media content whose orientation has been modifiedto align with the field of view of the user, such that the one or moremedia content is presented as a virtual overlay in relation to the atleast a portion of the printed publication.

In some embodiments, the apparatus might comprise one of a VR headset,an AR headset, a set of AR glasses, a tablet computer, a set of bioniccontact lenses, a smart phone adapted as part of a VR headset, or asmart phone adapted as part of an AR system, and/or the like. In somecases, the apparatus might further comprise a camera that captures oneor more images of eyes of the user as the user is looking toward the atleast a portion of the printed publication, and determining, with thecomputer system, a field of view of a user comprises determining, withthe computer system, a field of view of a user based on an analysis ofthe captured one or more images of the eyes of the user.

Merely by way of example, in some cases, the printed publication mightcomprise one of a fiction novel, a non-fiction novel, a children's book,a newspaper, a magazine, a comic book, a comic strip, a graphic novel,an encyclopedia, a dictionary, a map, a calendar, a greeting card, abookmark, a venue or event ticket stub, or a poster, and/or the like. Insome instances, the at least one trigger might each comprise one of aRFID tag, a BLE tag, a NFC tag, or a RTLS tag, and/or the like.

In some cases, the one or more media content might comprise at least oneof a 3D still graphic, a 3D animated graphic, or a 3D interactiveanimated graphic, and/or the like.

In yet another aspect, a system might comprise a wireless communicationdevice, a computing system, and a visual output device. The wirelesscommunication device might comprise a transceiver, at least one firstprocessor, and a first non-transitory computer readable medium. Thefirst non-transitory computer readable medium might have stored thereoncomputer software comprising a first set of instructions that, whenexecuted by the at least one first processors, causes the image capturedevice to: establish communications between a visual output device andat least one trigger of one or more non-visual triggers associated witha printed publication.

The computing system might comprise at least one second processor and asecond non-transitory computer readable medium. The secondnon-transitory computer readable medium might have stored thereoncomputer software comprising a second set of instructions that, whenexecuted by the at least one second processors, causes the computingsystem to: in response to establishing communications between the visualoutput device and the at least one trigger, identify one or more mediacontent of a plurality of media content that are associated with the atleast one trigger; determine a field of view of a user, the field ofview including at least a portion of the printed publication; modify anorientation of the one or more media content to align with the field ofview of the user; and send the one or more media content whoseorientation has been modified to align with the field of view of theuser to the visual output device.

The visual output device might comprise a display device, at least onethird processor, and a third non-transitory computer readable medium.The third non-transitory computer readable medium might have storedthereon computer software comprising a third set of instructions that,when executed by the at least one third processors, causes the visualoutput device to: receive the one or more media content whoseorientation has been modified to align with the field of view of theuser from the computing system; and present, with the display device andto the user, the one or more media content whose orientation has beenmodified to align with the field of view of the user, such that the oneor more media content is presented as a virtual overlay in relation tothe at least a portion of the printed publication.

In another aspect, a method might comprise presenting, with a visualoutput device and to a user, one or more media content, such that theone or more media content is presented as a virtual overlay in relationto at least a portion of a physical object; and determining, with acomputing system, how the one or more media content should be presentedto the user. The method might also comprise, based on a determination topresent the one or more media content as a virtual reality presentation,and based on a determination that the one or more media content is notalready being presented as a virtual reality presentation, shifting,with the computing system, the presentation of the one or more mediacontent to a virtual reality presentation, and presenting, with thevisual output device and to the user, one or more second media content,the one or more second media content comprising virtual reality content.The method might further comprise, based on a determination to presentthe one or more media content as a mixed reality presentation, and basedon a determination that the one or more media content is not alreadybeing presented as a mixed reality presentation, shifting, with thecomputing system, the presentation of the one or more media content to amixed reality presentation, and presenting, with the visual outputdevice and to the user, one or more third media content, the one or morethird media content comprising mixed reality content.

In some embodiments, the visual output device might comprise one of a VRheadset, an AR headset, a set of AR glasses, a tablet computer, a set ofbionic contact lenses, a smart phone adapted as part of a VR headset, ora smart phone adapted as part of an AR system, and/or the like. In somecases, the physical object might comprise one of a fiction novel, anon-fiction novel, a children's book, a newspaper, a magazine, a comicbook, a comic strip, a graphic novel, an encyclopedia, a dictionary, amap, a calendar, a greeting card, a bookmark, a venue or event ticketstub, a poster, a Christmas tree bulb, a Christmas tree ornament, aHanukkah menorah, a gravestone, a sentimental artifact or ornamentassociated with the user, or a sentimental artifact or ornamentassociated with another individual who is associated with the user,and/or the like. In some instances, the one or more media content mightcomprise at least one of a 3D still graphic, a 3D animated graphic, or a3D interactive animated graphic, and/or the like.

The method, according to some embodiments, might further comprise atleast one of detecting, with one or more sensors, one or more physicalcharacteristics of the user in response to stimuli from presentation ofmedia content or receiving, with the computing system, a user inputindicating selection of how the media content should be presented. Insuch embodiments, determining, with the computing system, how the one ormore media content should be presented to the user might be based on atleast one of analysis of the detected one or more physicalcharacteristics of the user or the received user input. In someembodiments, the method might further comprise, based on a determinationto present the one or more media content as an augmented realitypresentation, and based on a determination that the one or more mediacontent is not already being presented as an augmented realitypresentation, shifting, with the computing system, the presentation ofthe one or more media content to an augmented reality presentation, andpresenting, with the visual output device and to the user, one or morefourth media content, the one or more fourth media content comprisingaugmented reality content.

In yet another aspect, an apparatus might comprise a display device, atleast one processor, and a non-transitory computer readable medium, thenon-transitory computer readable medium having stored thereon computersoftware comprising a set of instructions that, when executed by the atleast one processors, causes the apparatus to: present, with the displaydevice and to a user, one or more media content, such that the one ormore media content is presented as a virtual overlay in relation to atleast a portion of a physical object; determine how the one or moremedia content should be presented to the user; based on a determinationto present the one or more media content as a virtual realitypresentation, and based on a determination that the one or more mediacontent is not already being presented as a virtual realitypresentation, shift the presentation of the one or more media content toa virtual reality presentation, and present, with the display device andto the user, one or more second media content, the one or more secondmedia content comprising virtual reality content; and based on adetermination to present the one or more media content as a mixedreality presentation, and based on a determination that the one or moremedia content is not already being presented as a mixed realitypresentation, shift the presentation of the one or more media content toa mixed reality presentation, and present, with the display device andto the user, one or more third media content, the one or more thirdmedia content comprising mixed reality content.

In some embodiments, the apparatus might comprise one of a VR headset,an AR headset, a set of AR glasses, a tablet computer, a set of bioniccontact lenses, a smart phone adapted as part of a VR headset, or asmart phone adapted as part of an AR system, and/or the like. In somecases, the physical object might comprise one of a fiction novel, anon-fiction novel, a children's book, a newspaper, a magazine, a comicbook, a comic strip, a graphic novel, an encyclopedia, a dictionary, amap, a calendar, a greeting card, a bookmark, a venue or event ticketstub, a poster, a Christmas tree bulb, a Christmas tree ornament, aHanukkah menorah, a gravestone, a sentimental artifact or ornamentassociated with the user, or a sentimental artifact or ornamentassociated with another individual who is associated with the user,and/or the like. In some instances, the one or more media content mightcomprise at least one of a 3D still graphic, a 3D animated graphic, or a3D interactive animated graphic, and/or the like.

According to some embodiments, the apparatus might further comprise atleast one of one or more sensors or one or more user input devices. Theset of instructions, when executed by the one or more processors,further causes the apparatus to perform at least one of: detecting, withthe one or more sensors, one or more physical characteristics of theuser in response to stimuli from presentation of media content; orreceiving, from the one or more user input devices, a user inputindicating selection of how the media content should be presented. Insuch embodiments, determining how the one or more media content shouldbe presented to the user might be based on at least one of analysis ofthe detected one or more physical characteristics of the user or thereceived user input.

In some instances, the set of instructions, when executed by the one ormore processors, further causes the apparatus to: based on adetermination to present the one or more media content as an augmentedreality presentation, and based on a determination that the one or moremedia content is not already being presented as an augmented realitypresentation, shift the presentation of the one or more media content toan augmented reality presentation, and present, with the display deviceand to the user, one or more fourth media content, the one or morefourth media content comprising augmented reality content.

In still another aspect, a system might comprise a visual output deviceand a computing system. The visual output device might comprise adisplay device, at least one first processor, and a first non-transitorycomputer readable medium. The first non-transitory computer readablemedium might have stored thereon computer software comprising a firstset of instructions that, when executed by the at least one firstprocessors, causes the visual output device to: present, with thedisplay device and to a user, one or more media content, such that theone or more media content is presented as a virtual overlay in relationto at least a portion of a physical object.

The computing system might comprise at least one second processor and asecond non-transitory computer readable medium. The secondnon-transitory computer readable medium might have stored thereoncomputer software comprising a second set of instructions that, whenexecuted by the at least one second processors, causes the computingsystem to: determine how the one or more media content should bepresented to the user; based on a determination to present the one ormore media content as a virtual reality presentation, and based on adetermination that the one or more media content is not already beingpresented as a virtual reality presentation, shift the presentation ofthe one or more media content to a virtual reality presentation; andbased on a determination to present the one or more media content as amixed reality presentation, and based on a determination that the one ormore media content is not already being presented as a mixed realitypresentation, shift the presentation of the one or more media content toa mixed reality presentation.

The first set of instructions, when executed by the at least one firstprocessors, might further cause the visual output device to: based on adetermination to present the one or more media content as a virtualreality presentation, and based on a determination that the one or moremedia content is not already being presented as a virtual realitypresentation, present, with the display device and to the user, one ormore second media content, the one or more second media contentcomprising virtual reality content; and based on a determination topresent the one or more media content as a mixed reality presentation,and based on a determination that the one or more media content is notalready being presented as a mixed reality presentation, present, withthe display device and to the user, one or more third media content, theone or more third media content comprising mixed reality content.

In another aspect, a method might comprise determining, with a computingsystem, how at least one of one or more first media content of aplurality of media content or one or more captured images of one or morephysical objects should be presented to a user. The method might furthercomprise, based on a determination to present the one or more firstmedia content as a virtual reality presentation, and based on adetermination that the one or more first media content is not alreadybeing presented as a virtual reality presentation, controlling, with thecomputing system via a first API, an asset integration and controlsystem to cause the asset integration and control system to modify thefirst media content to generate one or more second media content, theone or more second media content comprising virtual reality content, andsending, with the computing system via a second API, the one or moresecond media content to a first visual output device of a plurality ofvisual output devices that presents the one or more second media contentto the user.

The method might also comprise, based on a determination to present theone or more first media content and the one or more captured images ofthe one or more physical objects as an augmented reality presentation,and based on a determination that the one or more first media contentand the one or more captured images of the one or more physical objectsare not already being presented as an augmented reality presentation,controlling, with the computing system via the first API, the assetintegration and control system to cause the asset integration andcontrol system to generate one or more third media content by modifyingthe one or more first media content as a virtual overlay in relation tothe one or more captured images of the one or more physical objects, theone or more third media content comprising augmented reality content,and sending, with the computing system via the second API, the one ormore third media content to the first visual output device that presentsthe one or more third media content to the user. The method mightfurther comprise, based on a determination to present the one or morefirst media content and the one or more captured images of the one ormore physical objects as a mixed reality presentation combining elementsof virtual reality and augmented reality, and based on a determinationthat the one or more first media content and the one or more capturedimages of the one or more physical objects are not already beingpresented as a mixed reality presentation, controlling, with thecomputing system via the first API, the asset integration and controlsystem to cause the asset integration and control system to generate oneor more fourth media content based on the one or more first mediacontent and the one or more captured images of the one or more physicalobjects, the one or more fourth media content comprising mixed realitycontent, and sending, with the computing system via the second API, theone or more fourth media content to the first visual output device thatpresents the one or more fourth media content to the user.

In some embodiments, the method might further comprise tracking, withthe computing system, a real-time status of each of the plurality of themedia content in each of at least one of the asset integration andcontrol system via the first API, the first visual output device via thesecond API, an asset production system via a third API, and a productionmanagement system via a fourth API. In some cases, the method might alsocomprise distributing, with the computing system, at least one fifthmedia content of the plurality of the media content whose statusindicate it is ready for integration from the asset production system tothe asset integration and control system and distributing, with thecomputing system, at least one sixth media content of the plurality ofthe media content whose status indicate it is ready for distributionfrom the asset integration and control system to the first visual outputdevice. In some instances, the method might further comprise syncing,with the computing system, the at least one sixth media content with oneor more second visual output devices of the plurality of visual outputdevices, the one or more second visual output devices being separatefrom the first visual output device and, in response to receivingupdates to the at least one sixth media content, sending, with thecomputing system, the updates to the at least one sixth media content toeach of the first visual output device and each of the one or moresecond visual output devices.

According to some embodiments, the method might further comprisereceiving, with the computing system, the one or more first mediacontent from an asset production system via a third API and receiving,with the computing system, the one or more captured images of the one ormore physical objects from an image capture device. In some embodiments,the method might further comprise determining, with the computingsystem, that the one or more first media content is ready fordistribution to the first visual output device; sending, with thecomputing system, one or more first messages to a production managementsystem via a fourth API, wherein the one or more first messages causesthe production management system to change a status of the one or morefirst media content to indicate that the one or more first media contentis ready for distribution to the first visual output device; andsending, with the computing system, one or more second messages to theasset production system via the third API, wherein the one or moresecond messages causes the asset production system to send the one ormore first media content to the computing system via the third API.

In some cases, the method might further comprise determining, with thecomputing system, that one or more second media content is not ready fordistribution to the first visual output device; sending, with thecomputing system, one or more third messages to a production managementsystem via a fourth API, wherein the one or more third messages causesthe production management system to change a status of the one or moresecond media content to indicate that the one or more second mediacontent is not ready for distribution to the first visual output device;and sending, with the computing system, one or more fourth messages toan asset production system via the third API, wherein the one or morefourth messages causes the asset production system to send the one ormore fifth messages to one or more developers to continue work on theone or more second media content.

In some instances, the computing system might comprise at least one of aserver computer, a cloud computing system, or a computing system of thevisual output device, and/or the like. According to some embodiments,the plurality of visual output devices might each comprise one of a VRheadset, an AR headset, a set of AR glasses, a tablet computer, a set ofbionic contact lenses, a smart phone adapted as part of a VR headset, ora smart phone adapted as part of an AR system, and/or the like.

In yet another aspect, a computing system might comprise at least oneprocessor and a non-transitory computer readable medium, thenon-transitory computer readable medium having stored thereon computersoftware comprising a set of instructions that, when executed by the atleast one processors, causes the computing system to: determine how atleast one of one or more first media content of a plurality of mediacontent or one or more captured images of one or more physical objectsshould be presented to a user; based on a determination to present theone or more first media content as a virtual reality presentation, andbased on a determination that the one or more first media content is notalready being presented as a virtual reality presentation, control, viaa first API, an asset integration and control system to cause the assetintegration and control system to modify the first media content togenerate one or more second media content, the one or more second mediacontent comprising virtual reality content, and send, via a second API,the one or more second media content to a first visual output device ofa plurality of visual output devices that presents the one or moresecond media content to the user; based on a determination to presentthe one or more first media content and the one or more captured imagesof the one or more physical objects as an augmented realitypresentation, and based on a determination that the one or more firstmedia content and the one or more captured images of the one or morephysical objects are not already being presented as an augmented realitypresentation, control, via the first API, the asset integration andcontrol system to cause the asset integration and control system togenerate one or more third media content by modifying the one or morefirst media content as a virtual overlay in relation to the one or morecaptured images of the one or more physical objects, the one or morethird media content comprising augmented reality content, and send, viathe second API, the one or more third media content to the first visualoutput device that presents the one or more third media content to theuser; and based on a determination to present the one or more firstmedia content and the one or more captured images of the one or morephysical objects as a mixed reality presentation combining elements ofvirtual reality and augmented reality, and based on a determination thatthe one or more first media content and the one or more captured imagesof the one or more physical objects are not already being presented as amixed reality presentation, control, via the first API, the assetintegration and control system to cause the asset integration andcontrol system to generate one or more fourth media content based on theone or more first media content and the one or more captured images ofthe one or more physical objects, the one or more fourth media contentcomprising mixed reality content, and send, via the second API, the oneor more fourth media content to the first visual output device thatpresents the one or more fourth media content to the user.

Various modifications and additions can be made to the embodimentsdiscussed without departing from the scope of the invention. Forexample, while the embodiments described above refer to particularfeatures, the scope of this invention also includes embodiments havingdifferent combination of features and embodiments that do not includeall of the above described features.

Specific Exemplary Embodiments

We now turn to the embodiments as illustrated by the drawings. FIGS. 1-9illustrate some of the features of the method, system, and apparatus forimplementing computer-simulated reality interactions, and, in particularembodiments, to methods, systems, and computer software for implementingcomputer-simulated reality interactions between users and physicalpublications, as referred to above. FIGS. 1-5 are directed to themethods, systems, and apparatuses for implemented computer-simulatedreality user experiences, while FIGS. 6 and 7 are directed to themethods, systems, and apparatuses for implemented computer-simulatedreality platform, and FIGS. 8 and 9 are directed to the hardware systemsfor implementing the computer-simulated reality user experiences and thecomputer-simulated reality platform. The methods, systems, andapparatuses illustrated by FIGS. 1-9 refer to examples of differentembodiments that include various components and steps, which can beconsidered alternatives, or which can be used in conjunction with oneanother in the various embodiments. The description of the illustratedmethods, systems, and apparatuses shown in FIGS. 1-9 is provided forpurposes of illustration and should not be considered to limit the scopeof the different embodiments.

Computer-Simulated Reality UX

With reference to the figures, FIG. 1 is a schematic diagramillustrating a system 100 for implementing computer-simulated realityinteractions between users and physical publications, in accordance withvarious embodiments.

In the non-limiting embodiment of FIG. 1, system 100 might comprise oneor more user devices 105 (also referred to herein as “visual outputdevices” or the like). Although specific embodiments of user devices 105are shown in FIG. 1 (e.g., a tablet computer, a smart phone, and avirtual reality or augmented reality headset), the various embodimentsare not so limited, and each user device 105 might include, withoutlimitation, one of a VR headset, an a AR headset, a set of AR glasses, atablet computer, a set of bionic contact lenses, a smart phone adaptedas part of a VR headset, or a smart phone adapted as part of an ARsystem, and/or the like. In some embodiments, system 100 might furthercomprise a computing system 110 (including, without limitation, a servercomputer, a cloud computing system, and/or the like) that is separatefrom, or remote from, the one or more user devices 105, and that isaccessible via network 115 over a wired connection (e.g., as shown inFIG. 1 by the solid line between the one or more user devices 105 andthe network 115) or over a wireless connection (e.g., as shown in FIG. 1by the lightning bolt symbol between the one or more user devices 105and the network 115). In some cases, the network 115 might include, butis not limited to, a local area network (“LAN”), including, withoutlimitation, a fiber network, an Ethernet network, a Token-Ring™ network,and/or the like; a wide-area network (“WAN”); a wireless wide areanetwork (“WWAN”); a virtual network, such as a virtual private network(“VPN”); the Internet; an intranet; an extranet; a public switchedtelephone network (“PSTN”); an infra-red network; a wireless network,including, without limitation, a network operating under any of the IEEE802.11 suite of protocols, the Bluetooth™ protocol known in the art,and/or any other wireless protocol; and/or any combination of theseand/or other networks. In a particular embodiment, the network mightinclude an access network of the service provider (e.g., an Internetservice provider (“ISP”)). In another embodiment, the network mightinclude a core network of the service provider, and/or the Internet.

According to some embodiments, system 100 might further comprise aprinted publication 125. Although a specific embodiment of the printedpublication 125 is shown in FIG. 1 (e.g., a book), the variousembodiments are not so limited and the printed publication 125 caninclude, without limitation, one of a fiction novel, a non-fictionnovel, a children's book, a newspaper, a magazine, a comic book, a comicstrip, a graphic novel, an encyclopedia, a dictionary, a map, acalendar, a greeting card, a bookmark, a venue or event ticket stub, ora poster, and/or the like. In some embodiments, one or more triggers 130might be provided on at least a portion of the printed publication, andmight each include, but is not limited to, one of a QR code provided onthe at least a portion of the publication, a barcode provided on the atleast a portion of the publication, a graphical code provided on asticker affixed to the at least a portion of the publication, a graphicprinted on the at least a portion of the publication, a keyword or keycode printed on the at least a portion of the publication, a cover imageof the printed publication, a title printed on a cover of the printedpublication, author name printed on the cover of the printedpublication, a stuffed character associated with the printed publicationthat is positioned in proximity to the printed publication, a 3D-printedcharacter associated with the printed publication that is positioned inproximity to the printed publication, or a 3D-printed object associatedwith the printed publication that is positioned in proximity to theprinted publication, and/or the like. According to some embodiments, theuser device 105 might further include, but is not limited to, a firstcamera 135 (as shown in the side view of user device 105 as shown alongthe direction of arrows A-A of the front view of the user device 105), asecond camera 140, and a display device or display screen 145, and/orthe like.

In operation, the first camera 135 of user device 105 might capture oneor more images of at least a portion of the printed publication 125(e.g., a front cover, the entire publication, a back cover, a page ofthe printed publication, a binding of the printed publication, a portionof any of these, and/or the like). The user device 105 and/or thecomputing system 110 might analyze the captured one or more images ofthe at least a portion of the printed publication to determine whetherthe captured one or more images contain one or more triggers 130 (asdescribed above). According to some embodiments, the user device 105 orthe computing system 110 might map points on the one or more capturedimages of the at least a portion of the printed publication 125 as partof the analysis to determine whether the captured one or more imagescontain one or more triggers 130.

Based on a determination that the captured one or more images contain atleast one trigger of the one or more triggers, the user device 105and/or the computing system 110 might identify one or more media content160 of a plurality of media content that are associated with the atleast one trigger. The user device 105 and/or the computing system 110might subsequently determine a field of view of a user (e.g., bycapturing an image of an eye(s) of the user using second camera 140),the field of view including at least a portion of the printedpublication, and might modify an orientation of the one or more mediacontent to align with the field of view of the user, if necessary (i.e.,based on a determination that the orientation of the one or more mediacontent is not already aligned with the field of view of the user, orthe like). The user device 105 and/or the computing system 110 mightthen display, on display device 145 of user device 105 the one or moremedia content (whose orientation may have been modified to align withthe field of view of the user), such that the one or more media contentis presented as a virtual overlay in relation to the at least a portionof the printed publication. In the non-limiting embodiment of FIG. 1,the one or more media content 160 comprises a character in the act ofthrowing an object (e.g., a snowball or the like), and the character ispresented as a virtual overlay on top of an image of the cover of theprinted publication 125. Although the one or more media content 160 isdepicted as being displayed on the display device of the tablet computer105 in FIG. 1, the various embodiments are not so limited, and the oneor more media content 160 may be displayed on the display device of anyof the user devices 105 (e.g., a VR headset, an AR headset, a set of ARglasses, a set of bionic contact lenses, a smart phone adapted as partof a VR headset, or a smart phone adapted as part of an AR system,and/or the like) as described above.

In some cases, system 100 might further comprise one or more databases120 that store at least one of information regarding the one or moretriggers 130 associated with each printed publication 125 (among aplurality of printed publications), information regarding the one ormore media content 160 that are associated with each printed publication125 (among the plurality of printed publications), the one or more mediacontent 160 themselves, formatting information for each of the visualoutput devices or display devices of user devices 105, and/or the like.

According to some embodiments, the functionalities of the one or moreuser devices 105 as described above might be provided by a softwareapplication (“app”) that are downloaded and installed on the one or moreuser devices 105, and the software app provides the functionalities forcapturing images of the at least a portion of the printed publication,analyzing the images for triggers, identifying media content associatedwith the triggers, determining the field of view of the user, modifying(if necessary) the orientation of the media content to align with thefield of view of the user, and presenting the media content to the useras a virtual overlay in relation to (in some cases, an image of) the atleast a portion of the printed publication.

In some embodiments, the trigger might include a particular arrangementof two or more printed publications. For example, placing two books in aparticular series of books next to each other might serve as a triggerfor generating media content comprising characters from the two booksthat are presented as virtual overlays in relation to the two books,where the characters are presented as interacting with each other (e.g.,a first character initiates a snowball fight with a second character,where each character might be presented as overlays on top of one of thetwo books, or the like). In another example, placing three or more booksin a series of books in a first sequence or order relative to each othermight serve as a trigger for one set of media content (which mightinclude a first set of interactions amongst characters in the series ofbooks), while placing the three or more books in a second sequence ororder relative to each other might serve as a trigger for another set ofmedia content (which might include a second set of interactions amongstcharacters in the series of books). In some cases, rather thancharacters, the arrangement of the printed publications in a particularorder might result in triggering presentation of a map, clues, or othercontent that might lead users to actual or virtual prizes or the like(e.g., as part of a marketing campaign for the series of books, for amovie, for a TV series, for a game, for toys, and/or the like). In somecases, placing all of the books in a series together in a particularconfiguration and/or order might serve as a trigger for generating avirtual location or venue (e.g., presenting a virtual representation ofa town, a town center, a city, a city center, a building, a room, etc.,that is associated with the book series or the like). These virtualprojections described above (also referred to herein as “combinedeffects”) might be 3D virtual projections, or 2D slices that mightchange when the user device moves around or relative to the printedpublications (e.g., looking from one angle might allow the user to seeone side of the virtual projection, while looking from another anglemight allow the user to see another side of the same virtual projection,or the like).

Alternatively, or additionally, geolocation functionalities of userdevices 105 might be used as an additional trigger for each of thepresentation of media content. For example, if the user is located in alocation that is associated with one of the settings of the story of theprinted publication and implements the app as described above, theresultant media content might be specially tailored for that locationcompared with media content that would otherwise be triggered.

Alternatively, or additionally, weather tracking functionalities of userdevices 105 might be used as an additional trigger (which may also becoupled with geolocation functionalities) for triggering presentation ofparticular media content. For example, if the user device (through itsweather tracking apps or the Internet and/or with information about thegeolocation of the user device) determines that it is snowing at thelocation of the user device as the user is using the app as describedabove, the resultant media content being presented to the user inresponse to the trigger associated with the printed publication mightinclude characters associated the printed publication engaging in asnowball fight or other snow sports, consistent with the story. If theuser device otherwise determines that it is sunny at the location of theuser device as the user is using the app as described above, theresultant media content being presented to the user in response to thetrigger associated with the printed publication might include charactersassociated the printed publication engaging in a water balloon fight orother summer activities, consistent with the story. And so on.

According to some embodiments, the media content being presented as avirtual overlay might include characters that interact with the user, asin a conversation or a game, and/or the like. For example, such virtualconversations between the projected virtual character and the user mightbe consistent with conversations or topics associated with the story ofthe printed publication. In the case of interactive games, the virtualcharacter might engage in a snowball fight or water balloon fight withthe user and the user tapping or otherwise entering inputs on the userdevice (e.g., touching particular buttons or portions of the screen)might cause virtual snowballs or water balloons to fly toward thevirtual character, and so on.

Although the above embodiments are described with respect to printedpublications, in some cases, e-books or books read from a computingdevice might be used instead of the printed publications. In such cases,a separate user device as described above might capture images of thee-book cover or page to determine if the images contain triggers, andthe other processes would otherwise be the same or similar. Rather thanputting two or more printed publications together to trigger combinedeffects as described above, the e-book reader or computing device mightdisplay the covers of the two or more printed publications on the samescreen. In some cases, the app might also serve as an e-book reader, andmight allow for arrangement of multiple publications on the displayscreen to trigger the combined effects. In alternative cases, ratherthan using a separate user device to capture images of the e-books, theapp might allow for reading the e-books while providing options fortriggering the processes described above without using a camera (perhapsinstead using screen capture technology to capture images of the screenshowing the e-publications or the like, or simply recognizing triggersthat are being displayed on the screen). In some cases, triggers mightbe flagged for the user to actuate by clicking on the triggers,resulting in presentation of the triggered media content. According tosome embodiments, a 3D projection or holographic device or the likemight be used in conjunction with the e-book reader/user device topresent a 3D projection above the screen of the device (which mightallow a user wearing corresponding 3D/holographic glasses to view theprojections or which might allow glass-less 3D holographic projection ofthe media content).

In some embodiments, a graphic in the printed publication might serve asa trigger. For example, a map illustration might trigger a 3D projectionof the location (or one of the locations) shown by the map. In somecases, a map of a valley might trigger a 3D animation of a bird swoopingdown into the valley, or the like. A 2D map of a region of space mighttrigger a 3D pan of the same region of space. And so on. A 2Dillustration of a scene in the book might trigger a 3D animation of thesame scene. In these 3D projections, the user can change the point ofview by moving the display device to view a different angle of the 3Dprojections.

These interactions and media content as described are merelynon-limiting examples consistent with the embodiments of the invention,and other interactions and media content may be implemented (so long asconsistent with the embodiments described or implied herein).

FIG. 2 is a flow diagram illustrating a method 200 for implementingcomputer-simulated reality interactions between users and physicalpublications, in accordance with various embodiments.

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method illustrated by FIG. 2can be implemented by or with (and, in some cases, are described belowwith respect to) the system 100 of FIG. 1 (or components thereof), suchmethod may also be implemented using any suitable hardware (or software)implementation. Similarly, while the system 100 of FIG. 1 (or componentsthereof), can operate according to the method illustrated by FIG. 2(e.g., by executing instructions embodied on a computer readablemedium), the system 100 of FIG. 1 can also operate according to othermodes of operation and/or perform other suitable procedures.

With reference to FIG. 2, method 200, according to some embodiments,might comprise, at block 205, capturing, with a first camera, one ormore images of at least a portion of a printed publication (e.g.,printed publication 125 of FIG. 1 or the like). At block 210, method 200might comprise analyzing, with a computing system (which mightcorrespond to either user device 105 or computing system 110 of FIG. 1,or the like), the captured one or more images of the at least a portionof the printed publication to determine whether the captured one or moreimages contain one or more triggers (e.g., one or more triggers 130 ofFIG. 1, or the like). In some embodiments, the printed publication mightinclude, without limitation, one of a fiction novel, a non-fictionnovel, a children's book, a newspaper, a magazine, a comic book, a comicstrip, a graphic novel, an encyclopedia, a dictionary, a map, acalendar, a greeting card, a bookmark, a venue or event ticket stub, ora poster, and/or the like. The at least one trigger, in some cases,might each include, but is not limited to, one of a QR code provided onthe at least a portion of the publication, a barcode provided on the atleast a portion of the publication, a graphical code provided on asticker affixed to the at least a portion of the publication, a graphicprinted on the at least a portion of the publication, a keyword or keycode printed on the at least a portion of the publication, a cover imageof the printed publication, a title printed on a cover of the printedpublication, author name printed on the cover of the printedpublication, a stuffed character associated with the printed publicationthat is positioned in proximity to the printed publication, a 3D-printedcharacter associated with the printed publication that is positioned inproximity to the printed publication, or a 3D-printed object associatedwith the printed publication that is positioned in proximity to theprinted publication, and/or the like. In some cases, the first cameramight be part of or might be embodied in the user device. Alternatively,the first camera might be external to the user device, while beingcommunicatively coupled to the user device.

Method 200 might further comprise, based on a determination that thecaptured one or more images contain at least one trigger of the one ormore triggers, identifying, with the computing system, one or more mediacontent of a plurality of media content that are associated with the atleast one trigger (block 215). In some instances, the one or more mediacontent comprises at least one of a 3D still graphic, a 3D animatedgraphic, or a 3D interactive animated graphic, and/or the like, each ofwhich might include, without limitation, one of a graphic representationof one or more characters associated with the printed publication, anon-fictional geographical location associated with the printedpublication, a fictional geographical location associated with theprinted publication, a room associated with the printed publication, abuilding associated with the printed publication, a scene described inthe printed publication, a scene inspired by the scene described in theprinted publication, one or more items described in the printedpublication, or one or more items inspired by the one or more itemsdescribed in the printed publication, and/or the like.

At block 220, method 200 might comprise determining, with the computersystem, a field of view of a user, the field of view including at leasta portion of the printed publication. In some embodiments, determining afield of view of the user might comprise capturing, with a second camera(which might correspond to second camera 140 of user device 105 of FIG.1, or the like), one or more images of eyes of the user as the firstcamera captures the one or more images of the at least a portion of theprinted publication (optional block 225) and determining, with thecomputer system, a field of view of a user based on an analysis of thecaptured one or more images of the eyes of the user (e.g., relative tothe captured one or more images of the at least a portion of the printedpublication by the first camera, or the like) (optional block 230).

Method 200 might further comprise, at block 235, modifying, with thecomputer system, an orientation of the one or more media content toalign with the field of view of the user, if necessary (i.e., if it isdetermined that the orientation of one or more media content does notalready align with the field of view of the user). At block 240, method200 might comprise presenting, with a visual output device and to theuser, the one or more media content whose orientation has been modifiedto align with the field of view of the user, such that the one or moremedia content is presented as a virtual overlay in relation to the atleast a portion of the printed publication. Here, the visual outputdevice might correspond to the user device as described above, and mightinclude, without limitation, one of a VR headset, an AR headset, a setof AR glasses, a tablet computer, a set of bionic contact lenses, asmart phone adapted as part of a VR headset, or a smart phone adapted aspart of an AR system, and/or the like. According to some embodiments,presenting the one or more media content might comprise presenting, withthe visual output device and to the user, the one or more media contentwhose orientation has been modified to align with the field of view ofthe user, such that the one or more media content is at least one ofpresented as a virtual overlay above a cover of the printed publication,presented as a virtual overlay on a cover of the printed publication,presented as a virtual overlay beside the printed publication, presentedas a virtual overlay above a page of the printed publication, presentedas a virtual overlay on a page of the printed publication, presented asa virtual overlay in front of a surface of the printed publication,presented as a virtual overlay in front of a binding of the printedpublication, presented as a virtual overlay above a 2D graphic printedon a portion of the printed publication, or presented as a virtualoverlay on a 2D graphic printed on a portion of the printed publication,and/or the like.

FIG. 3 is a schematic diagram illustrating another system 300 forimplementing computer-simulated reality interactions between users andphysical publications, in accordance with various embodiments.

In the non-limiting embodiment of FIG. 3, system 300 might comprise oneor more user devices 305 (also referred to herein as “visual outputdevices” or the like). Although specific embodiments of user devices 305are shown in FIG. 3 (e.g., a tablet computer, a smart phone, and avirtual reality or augmented reality headset), the various embodimentsare not so limited, and each user device 305 might include, withoutlimitation, one of a VR headset, an AR headset, a set of AR glasses, atablet computer, a set of bionic contact lenses, a smart phone adaptedas part of a VR headset, or a smart phone adapted as part of an ARsystem, and/or the like. In some embodiments, system 300 might furthercomprise a computing system 310 (including, without limitation, a servercomputer, a cloud computing system, and/or the like) that is separatefrom, or remote from, the one or more user devices 305, and that isaccessible via network 315 over a wired connection (e.g., as shown inFIG. 3 by the solid line between the one or more user devices 305 andthe network 315) or over a wireless connection (e.g., as shown in FIG. 3by the lightning bolt symbol between the one or more user devices 305and the network 315). In some cases, the network 315 might include, butis not limited to, a LAN, including, without limitation, a fibernetwork, an Ethernet network, a Token-Ring⋅“′” network, and/or the like;a WAN; a WWAN; a virtual network, such as a VPN; the Internet; anintranet; an extranet; a PSTN; an infra-red network; a wireless network,including, without limitation, a network operating under any of the IEEE802.11 suite of protocols, the Bluetooth™ protocol known in the art,and/or any other wireless protocol; and/or any combination of theseand/or other networks. In a particular embodiment, the network mightinclude an access network of the service provider (e.g., an ISP). Inanother embodiment, the network might include a core network of theservice provider, and/or the Internet.

According to some embodiments, system 300 might further comprise aprinted publication 325. Although a specific embodiment of the printedpublication 325 is shown in FIG. 3 (e.g., a book), the variousembodiments are not so limited and the printed publication 325 caninclude, without limitation, one of a fiction novel, a non-fictionnovel, a children's book, a newspaper, a magazine, a comic book, a comicstrip, a graphic novel, an encyclopedia, a dictionary, a map, acalendar, a greeting card, a bookmark, a venue or event ticket stub, ora poster, and/or the like. In some embodiments, one or more triggers 330might be provided on at least a portion of the printed publication, andmight each comprise a non-visual trigger, including, but is not limitedto, one of a RFID tag, a BLE tag, a NFC tag, or a RTLS tag, and/or thelike. According to some embodiments, the user device 305 might furtherinclude, but is not limited to, a transceiver 335 that wirelesslycommunicates with the one or more triggers 330 (e.g., as shown in FIG. 3by the lightning bolt symbol between the transceiver 335 and the one ormore triggers 330), a camera 340, a display device or display screen345, one or more processors 350, memory 355, and/or the like.

In operation, the user device 305 (or perhaps the transceiver 335) mightdetermine whether the user device 305 is within communications range ofat least one non-visual trigger 330 associated with the printedpublication. Based on a determination that the user device 330 is withincommunications range of at least one non-visual trigger 330, wirelesscommunications may be established between the transceiver 335 and the atleast one non-visual trigger 330 associated with the printedpublication.

In response to establishing communications between the user device 305and the at least one non-visual trigger 330, the user device 305 and/orthe computing system 310 might identify one or more media content 360 ofa plurality of media content that are associated with the at least onetrigger. The (processor 350 of the) user device 305 and/or the computingsystem 310 might subsequently determine a field of view of a user (e.g.,by capturing an image of an eye(s) of the user using camera 340), thefield of view including at least a portion of the printed publication,and might modify an orientation of the one or more media content toalign with the field of view of the user, if necessary (i.e., based on adetermination that the orientation of the one or more media content isnot already aligned with the field of view of the user, or the like).The user device 305 and/or the computing system 310 might then display,on display device 345 of user device 305 the one or more media content(whose orientation may have been modified to align with the field ofview of the user), such that the one or more media content is presentedas a virtual overlay in relation to the at least a portion of theprinted publication. In the non-limiting embodiment of FIG. 3, the oneor more media content 360 comprises a character in the act of throwingan object (e.g., a snowball or the like), and the character is presentedas a virtual overlay on top of an image of the cover of the printedpublication 325. Although the one or more media content 360 is depictedas being displayed on the display device of the tablet computer 305 inFIG. 3, the various embodiments are not so limited, and the one or moremedia content 360 may be displayed on the display device of any of theuser devices 305 (e.g., a VR headset, an AR headset, a set of ARglasses, a set of bionic contact lenses, a smart phone adapted as partof a VR headset, or a smart phone adapted as part of an AR system,and/or the like) as described above.

In some cases, system 300 might further comprise one or more databases320 that store at least one of information regarding the one or moretriggers 330 associated with each printed publication 325 (among aplurality of printed publications), information regarding the one ormore media content 360 that are associated with each printed publication325 (among the plurality of printed publications), the one or more mediacontent 360 themselves, formatting information for each of the visualoutput devices or display devices of user devices 305, and/or the like.

According to some embodiments, the functionalities of the one or moreuser devices 305 as described above might be provided by an app that aredownloaded and installed on the one or more user devices 305, and thesoftware app provides the functionalities for establishingcommunications between (the transceiver 335 of) the user device and thenon-visual triggers, identifying media content associated with thenon-visual triggers, determining the field of view of the user,modifying (if necessary) the orientation of the media content to alignwith the field of view of the user, and presenting the media content tothe user as a virtual overlay in relation to (in some cases, an imageof) the at least a portion of the printed publication.

In some embodiments, the trigger might include a particular arrangementof two or more printed publications. For example, placing two books in aparticular series of books next to each other, where establishingwireless communications between the transceiver and each triggerassociated with each book in turn, might serve as a trigger forgenerating media content comprising characters from the two books thatare presented as virtual overlays in relation to the two books, wherethe characters are presented as interacting with each other (e.g., afirst character initiates a snowball fight with a second character,where each character might be presented as overlays on top of one of thetwo books, or the like). In another example, placing three or more booksin a series of books in a first sequence or order relative to eachother, where establishing wireless communications between thetransceiver and each trigger associated with each book in turn, mightserve as a trigger for one set of media content (which might include afirst set of interactions amongst characters in the series of books),while placing the three or more books in a second sequence or orderrelative to each other, where establishing wireless communicationsbetween the transceiver and each trigger associated with each book inturn, might serve as a trigger for another set of media content (whichmight include a second set of interactions amongst characters in theseries of books). In some cases, rather than characters, the arrangementof the printed publications in a particular order might result intriggering presentation of a map, clues, or other content that mightlead users to actual or virtual prizes or the like (e.g., as part of amarketing campaign for the series of books, for a movie, for a TVseries, for a game, for toys, and/or the like). In some cases, placingall of the books in a series together in a particular configurationand/or order, where establishing wireless communications between thetransceiver and each trigger associated with each book either in turn orconcurrently, might serve as a trigger for generating a virtual locationor venue (e.g., presenting a virtual representation of a town, a towncenter, a city, a city center, a building, a room, etc., that isassociated with the book series or the like). These virtual projectionsdescribed above (also referred to herein as “combined effects”) might be3D virtual projections, or 2D slices that might change when the userdevice moves around or relative to the printed publications (e.g.,looking from one angle might allow the user to see one side of thevirtual projection, while looking from another angle might allow theuser to see another side of the same virtual projection, or the like).

Alternatively, or additionally, geolocation functionalities of userdevices 305 might be used as an additional trigger for each of thepresentation of media content. For example, if the user is located in alocation that is associated with one of the settings of the story of theprinted publication and implements the app as described above, theresultant media content might be specially tailored for that locationcompared with media content that would otherwise be triggered.

Alternatively, or additionally, weather tracking functionalities of userdevices 305 might be used as an additional trigger (which may also becoupled with geolocation functionalities) for triggering presentation ofparticular media content. For example, if the user device (through itsweather tracking apps or the Internet and/or with information about thegeolocation of the user device) determines that it is snowing at thelocation of the user device as the user is using the app as describedabove, the resultant media content being presented to the user inresponse to the trigger associated with the printed publication mightinclude characters associated the printed publication engaging in asnowball fight or other snow sports, consistent with the story. If theuser device otherwise determines that it is sunny at the location of theuser device as the user is using the app as described above, theresultant media content being presented to the user in response to thetrigger associated with the printed publication might include charactersassociated the printed publication engaging in a water balloon fight orother summer activities, consistent with the story. And so on.

According to some embodiments, the media content being presented as avirtual overlay might include characters that interact with the user, asin a conversation or a game, and/or the like. For example, such virtualconversations between the projected virtual character and the user mightbe consistent with conversations or topics associated with the story ofthe printed publication. In the case of interactive games, the virtualcharacter might engage in a snowball fight or water balloon fight withthe user and the user tapping or otherwise entering inputs on the userdevice (e.g., touching particular buttons or portions of the screen)might cause virtual snowballs or water balloons to fly toward thevirtual character, and so on.

Although the above embodiments are described with respect to printedpublications, in some cases, e-books or books read from a computingdevice might be used instead of the printed publications. In such cases,a separate user device as described above might establish wirelesscommunications with the e-book reader or computing device, and the otherprocesses would otherwise be the same or similar. In other words, ratherthan putting two or more printed publications together to triggercombined effects as described above, the e-book reader or computingdevice might display the covers of the two or more printed publicationson the same screen, and the e-book reader or computing device might havea transceiver that might simulate or serve as the non-visual trigger. Insome cases, the app might also serve as an e-book reader, and mightallow for arrangement of multiple publications on the display screen,while causing the e-book reader or computing device to use itstransceiver as a non-visual trigger associated with the publication totrigger the combined effects. In alternative cases, rather than using aseparate user device to capture images of the e-books, the app mightallow for reading the e-books while providing options for triggering theprocesses described above without using external transceivers or thelike (but rather causing a simulated or virtual trigger event to occurin lieu of the non-visual triggers described above). In some cases,virtual triggers might be flagged for the user to actuate by clicking onthe triggers, resulting in presentation of the triggered media content.According to some embodiments, a 3D projection or holographic device orthe like might be used in conjunction with the e-book reader/user deviceto present a 3D projection above the screen of the device (which mightallow a user wearing corresponding 3D/holographic glasses to view theprojections or which might allow glass-less 3D holographic projection ofthe media content).

In some embodiments, with e-book readers, accompanying audio readingsmay be utilized in conjunction with the user reading the e-book readers,and some triggers might comprise audio triggers, including, but notlimited to, audio key words associated with the text, sub-vocal keywordsas a hidden track underneath the audio track, audible tones, audibletunes, sub-audible tones, sub-audible tunes, etc. For example, after theaudio track has described a particular scene in the publication, theaudio track might pause as a 3D animation of the scene is projected andplayed out, and so on.

These interactions and media content as described are merelynon-limiting examples consistent with the embodiments of the invention,and other interactions and media content may be implemented (so long asconsistent with the embodiments described or implied herein).

System 300 of FIG. 3 might otherwise be similar, if not identical, tosystem 100 of FIG. 1, and the description of the components of system100 might similarly apply to corresponding components of system 300.

FIG. 4 is a flow diagram illustrating another method 400 forimplementing computer-simulated reality interactions between users andphysical publications, in accordance with various embodiments.

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method illustrated by FIG. 4can be implemented by or with (and, in some cases, are described belowwith respect to) the system 300 of FIG. 3 (or components thereof), suchmethod may also be implemented using any suitable hardware (or software)implementation. Similarly, while the system 300 of FIG. 3 (or componentsthereof), can operate according to the method illustrated by FIG. 4(e.g., by executing instructions embodied on a computer readablemedium), the system 300 of FIG. 3 can also operate according to othermodes of operation and/or perform other suitable procedures.

With reference to FIG. 4, method 400, according to some embodiments,might comprise, at block 405, establishing communications between avisual output device (which might correspond to user device 305 of FIG.3, or the like) and at least one trigger of one or more non-visualtriggers (which might correspond to the one or more triggers 330 of FIG.3, or the like) that are associated with a printed publication (e.g.,printed publication 325 of FIG. 3 or the like). In some embodiments,establishing communications between the visual output device and the atleast one trigger might comprise determining, with the computing system,whether the visual output device is within communications range of theone or more non-visual triggers associated with the printed publication(optional block 410) and, based on a determination that the visualoutput device is within communications range of the at least one triggerof the one or more non-visual triggers, establishing communicationsbetween the visual output device and the at least one trigger (optionalblock 415). In some embodiments, the printed publication might include,without limitation, one of a fiction novel, a non-fiction novel, achildren's book, a newspaper, a magazine, a comic book, a comic strip, agraphic novel, an encyclopedia, a dictionary, a map, a calendar, agreeting card, a bookmark, a venue or event ticket stub, or a poster,and/or the like. The at least one trigger, in some cases, might eachinclude, but is not limited to, one of a RFID tag, a BLE tag, a NFC tag,or a RTLS tag, and/or the like.

Method 400 might further comprise, in response to establishingcommunications between the visual output device and the at least onetrigger, identifying, with the computing system, one or more mediacontent of a plurality of media content that are associated with the atleast one trigger (block 420). In some instances, the one or more mediacontent might comprise at least one of a 3D still graphic, a 3D animatedgraphic, or a 3D interactive animated graphic, and/or the like, each ofwhich might include, without limitation, one of a graphic representationof one or more characters associated with the printed publication, anon-fictional geographical location associated with the printedpublication, a fictional geographical location associated with theprinted publication, a room associated with the printed publication, abuilding associated with the printed publication, a scene described inthe printed publication, a scene inspired by the scene described in theprinted publication, one or more items described in the printedpublication, or one or more items inspired by the one or more itemsdescribed in the printed publication, and/or the like.

At block 425, method 400 might comprise determining, with the computersystem, a field of view of a user, the field of view including at leasta portion of the printed publication. In some embodiments, determining afield of view of the user might comprise capturing, with a camera (whichmight correspond to camera 340 of user device 305 of FIG. 3, or thelike), one or more images of eyes of the user as the user is lookingtoward the at least a portion of the printed publication (optional block430) and determining, with the computer system, a field of view of auser based on an analysis of the captured one or more images of the eyesof the user (e.g., relative to the direction that the user is lookingtoward the at least a portion of the printed publication, or the like)(optional block 435).

Method 400 might further comprise, at block 440, modifying, with thecomputer system, an orientation of the one or more media content toalign with the field of view of the user, if necessary (i.e., if it isdetermined that the orientation of one or more media content does notalready align with the field of view of the user). At block 445, method400 might comprise presenting, with the visual output device and to theuser, the one or more media content whose orientation has been modifiedto align with the field of view of the user, such that the one or moremedia content is presented as a virtual overlay in relation to the atleast a portion of the printed publication. Here, the visual outputdevice might correspond to the user device as described above, and mightinclude, without limitation, one of a VR headset, an AR headset, a setof AR glasses, a tablet computer, a set of bionic contact lenses, asmart phone adapted as part of a VR headset, or a smart phone adapted aspart of an AR system, and/or the like. According to some embodiments,presenting the one or more media content might comprise presenting, withthe visual output device and to the user, the one or more media contentwhose orientation has been modified to align with the field of view ofthe user, such that the one or more media content is at least one ofpresented as a virtual overlay above a cover of the printed publication,presented as a virtual overlay on a cover of the printed publication,presented as a virtual overlay beside the printed publication, presentedas a virtual overlay above a page of the printed publication, presentedas a virtual overlay on a page of the printed publication, presented asa virtual overlay in front of a surface of the printed publication,presented as a virtual overlay in front of a binding of the printedpublication, presented as a virtual overlay above a 2D graphic printedon a portion of the printed publication, or presented as a virtualoverlay on a 2D graphic printed on a portion of the printed publication,and/or the like.

Method 400 of FIG. 4 might otherwise be similar, if not identical, tomethod 200 of FIG. 2, and the description of the processes of method 200might similarly apply to corresponding processes of method 400.

FIGS. 5A-5D (collectively, “FIG. 5”) are flow diagrams illustrating amethod 500 for implementing computer-simulated reality interactionsbetween users and physical objects, in accordance with variousembodiments. In FIG. 5, the process in method 500 continues from FIG. 5Ato FIG. 5B following the circular marker, denoted “A,” continues fromFIG. 5A to FIG. 5C following the circular marker, denoted “B,” continuesfrom FIG. 5A to FIG. 5D following the circular marker, denoted “C,” andreturns from each of FIGS. 5B-5D to FIG. 5A following the circularmarker, denoted, “D.”

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method illustrated by FIG. 5can be implemented by or with (and, in some cases, are described belowwith respect to) the system 100 of FIG. 1 (or components thereof) or thesystem 300 of FIG. 3 (or components thereof), such method may also beimplemented using any suitable hardware (or software) implementation.Similarly, while the system 100 of FIG. 1 (or components thereof) or thesystem 300 of FIG. 3 (or components thereof), can operate according tothe method illustrated by FIG. 3 (e.g., by executing instructionsembodied on a computer readable medium), the system 100 of FIG. 1 or thesystem 300 of FIG. 3 can also operate according to other modes ofoperation and/or perform other suitable procedures.

With reference to FIG. 5A, method 500 might comprise, at block 505,presenting, with a visual output device and to a user, one or more mediacontent, such that the one or more media content is presented as avirtual overlay in relation to at least a portion of a physical object.In some embodiments, the physical object might comprise a printedpublication, and the process at block 505 might correspond to theprocess at block 240 of method 200 of FIG. 2 (and might follow theprocesses at blocks 205-235 of method 200), or might correspond to theprocess at block 445 of method 400 of FIG. 4 (and might follow theprocesses at blocks 405-440 of method 400). In general, the physicalobject need not be a printed publication, and can include, withoutlimitation, one of a fiction novel, a non-fiction novel, a children'sbook, a newspaper, a magazine, a comic book, a comic strip, a graphicnovel, an encyclopedia, a dictionary, a map, a calendar, a greetingcard, a bookmark, a venue or event ticket stub, a poster, a Christmastree bulb, a Christmas tree ornament, a Hanukkah menorah, a gravestone,a sentimental artifact or ornament associated with the user, or asentimental artifact or ornament associated with another individual whois associated with the user, and/or the like. The visual output devicecan include, but is not limited to, one of a VR headset, an AR headset,a set of AR glasses, a tablet computer, a set of bionic contact lenses,a smart phone adapted as part of a VR headset, or a smart phone adaptedas part of an AR system, and/or the like. The one or more media contentcan include, without limitation, at least one of a 3D still graphic, a3D animated graphic, or a 3D interactive animated graphic, and/or thelike.

At optional block 510, method 500 might comprise detecting, with one ormore sensors, one or more physical characteristics of the user inresponse to stimuli from presentation of media content. In someinstances, the one or more sensors might include, without limitation, atleast one of one or more cameras, one or more heartrate monitors, one ormore fitness trackers, one or more microphones, and/or the like. The oneor more physical characteristics might include, but are not limited to,at least one of smiling, laughing, exclaiming, shouting, moving towardvirtual objects in the media content, moving away from virtual objectsin the media content, attempting to interact with virtual objects in themedia content, and/or the like. Alternatively or additionally, method500, at optional block 515, might comprise receiving, with the computingsystem, a user input indicating selection of how the media contentshould be presented. The user input, for example, might include, withoutlimitation, at least one of voice input, tactile input, attemptedtactile input, gesturing, eye movement selection of options, blinking,and/or the like.

Method 500 might further comprise, at block 520, determining, with acomputing system, how the one or more media content should be presentedto the user, which in some cases might be based on at least one ofanalysis of the detected one or more physical characteristics of theuser or the received user input. The process might continue to one ofthe process at block 525 in FIG. 5B following marker “A” (for virtualreality presentation), the process at block 535 in FIG. 5C followingmarker “B” (for mixed reality presentation), or the process at block 545in FIG. 5D following marker “C” (for augmented reality presentation).

With reference to FIG. 5B, method 500 might comprise, based on adetermination to present the one or more media content as a virtualreality presentation, and based on a determination that the one or moremedia content is not already being presented as a virtual realitypresentation, shifting, with the computing system, the presentation ofthe one or more media content to a virtual reality presentation (block525), and presenting, with the visual output device and to the user, oneor more second media content, the one or more second media contentcomprising virtual reality content (block 530). The process might returnto the process at optional block 510 in FIG. 5A.

Turning to FIG. 5C, method 500 might comprise, based on a determinationto present the one or more media content as a mixed realitypresentation, and based on a determination that the one or more mediacontent is not already being presented as a mixed reality presentation,shifting, with the computing system, the presentation of the one or moremedia content to a mixed reality presentation (block 535), andpresenting, with the visual output device and to the user, one or morethird media content, the one or more third media content comprisingmixed reality content (block 540). Here, “mixed reality” refers to acombination of augmented reality and virtual reality. For example, asimulated augmented reality presentation might be generated and embeddedwithin a virtual reality presentation. The process might return to theprocess at optional block 510 in FIG. 5A.

In FIG. 5D, method 500 might comprise, based on a determination topresent the one or more media content as an augmented realitypresentation, and based on a determination that the one or more mediacontent is not already being presented as an augmented realitypresentation, shifting, with the computing system, the presentation ofthe one or more media content to an augmented reality presentation(block 545), and presenting, with the visual output device and to theuser, one or more fourth media content, the one or more fourth mediacontent comprising augmented reality content (block 550). The processmight return to the process at optional block 510 in FIG. 5A.

In the embodiment of method 500, augmented reality is initiallytriggered, and subsequently serves as a bridge to one or mixed realityor virtual reality. The computing system (which in some cases is acomputer vision system, as described above) dynamically or selectivelyshifts between two of augmented reality, mixed reality, or virtualreality, in order to provide the user with an immersive interactiveexperience.

In some embodiments, the virtual reality presentation might includesimulated reality segments embedded within the virtual realitypresentation. In such cases, the user would be unable to distinguishbetween physical reality, augmented reality, and virtual reality. Forexample, a user might enter his or her physical or actual bedroom, inwhich a trigger might cause the system to generate an augmented realitypresentation of media, which might overlay the images of the bedroom.Further triggers or selections might cause a virtual simulation of thebedroom to be generated which might accurately reproduce an exact (ornear exact) replica of the bedroom (with the proper perspectives of theuser based on the positions and movements of the user's eyes relative topoints in the bedroom). The augmented reality presentation is thenshifted to a virtual reality presentation in which the simulated bedroomis seamlessly presented so that the user is unaware that the image ofthe physical bedroom has shifted to a simulated reproduction of thebedroom. In the virtual space, the simulated bedroom can be morphed ormanipulated in any suitable manner consistent with the story of thephysical object or printed publication that triggered the augmentedreality presentation. For instance, the story might include a scene inwhich the main character's bedroom morphs into a castle or into a spacevessel or into a fantastical landscape, and/or the like. In such a case,the user's simulated bedroom (which is very familiar to the user) wouldupon appropriate triggers morph into the castle, or space vessel, orfantastical landscape, and/or the like. In this manner, the seamlesstransition from physical or actual reality, to augmented reality, tovirtual reality would immerse the user more fully into the story.Further shifting between two of augmented reality, mixed reality, orvirtual reality at a time would facilitate the immersion process, thuslikely enhancing the user's cognitive and effective or affected empathyfor the story. In a similar manner, images of real people might bemorphed into characters in the story (and might either be completelychanged or might be partially changed such that some characteristics ofthe real people might remain after morphing in the virtual realm).

In some embodiments, where mirrors are used in the presentation, theuser might see himself or herself as the protagonist or as theantagonist. In such cases, a real mirror might be fully reproduced as asimulated mirror in which a morphed representation of the user might bepresented in a simulated reflection of the simulated mirror.Alternatively, the real mirror might be partially reproduced (the restbeing image of the actual mirror) such that a portion of the mirrorincludes a simulated reflection of the morphed representation of theuser. In this manner, the seamless transition from physical or actualreality, to augmented reality, to virtual reality would immerse the usermore fully into the story and might provide a first person point of viewthat enhances the user's sense of cognitive and effective or affectedempathy for the characters in the story. The presentation might thensimulate scenes in the story with the user playing the role of one ofthe characters in the story.

According to some embodiments, the user might be immersed, transported,or teleported into the story via the seamless transition from physicalor actual reality, to augmented reality, to virtual reality (or anycombination of such seamless transitions or shifts) to inhabit the roleof characters in the story. For example, in the story of Little RedRiding Hood, the user might find herself being transported or teleportedin this manner to inhabit the role of little Red Riding Hood as she goesoff to visit her grandma and as she encounters the wolf. Alternatively,or additionally, the. user might find herself being transported orteleported in the manner described above to inhabit the role of thegrandmother as she encounters the wolf, and what happens to her duringthe encounter. Alternatively, or additionally, the user might findherself being transported or teleported in the manner described above toinhabit the role of the wolf, as the wolf takes the role of the grandmaand as the wolf encounters Little Red Riding Hood. In another set ofexamples, e.g., in the story of the three little pigs, the user mightfind himself being transported or teleported in the manner describedabove to inhabit the role of one of the little pigs (or each in turn) asthe little pig builds its house (made of straw, sticks, or bricks),and/or might find himself being transported or teleported in the mannerdescribed above to inhabit the role of the wolf as it huffs, puffs, andblows in the straw and stick houses, and attempts to do so with thebrick house. In more sophisticated or more involved stories, such as theHarry Potter° series of books or the like, the user might find himselfor herself being transported or teleported in the manner described aboveto inhabit the role of the characters in the books, including, but notlimited to, Harry Potter®, Ron Weasley, Hermione Granger, LordVoldemort, Albus Dumbledore, Severus Snape, and/or the like.Collaborations might be made with author 3. K. Rowling to createexpanded storylines or scenes that allow the user to experiencebackstory or sidestory scenes with some of the secondary characters, tofurther immerse the user in the Potter universe. Similar userexperiences could be created for television-based stories,video-streaming service-based original stories (e.g., Netflix® originalseries, Amazon® original series, etc.), other movie-based stories (e.g.,Inception, the Marvel® Cinematic Universe series, the Star Wars® series,the Star Trek® series, etc.), comic book or manga stories (e.g., theMarvel® comics series, the DC® comics series, the Image comics series,etc.), stories in other novels, video game stories (e.g., the Star Wars®series of games, the Call of Duty® series of games, the Final Fantasyseries of games, the Mario Bros®. series of games, the Zelda® series ofgames, the Madden® series of games, Myst®, and so on) and/or the like.The system might adapt existing stories and scenes from the sourcestories into 3D immersive virtual/augmented/mixed reality environmentsand scenes or might (perhaps in collaboration with the original creatorsor licensed entities) to create new scenes and stories (that are thenadapted into 3D immersive virtual/augmented/mixed reality environmentsand scenes, etc.) based on these original source stories. In thesevarious user experiences, the users can learn what it is like toexperience certain trials and tribulations that the characters encounterin the stories, and to learn to perhaps see things from differentperspectives. In sum, the transitioning techniques described aboveenable a user to, in a sense, become particular characters, therebyenhancing the user's sense of cognitive and effective or affectedempathy for the characters in the story.

In some embodiments, aggregation, management, and retrieval of thevarious user experience settings and content can be performed orimplemented within the computer-simulated reality platform (which systemis described in detail below with respect to FIGS. 6 and 7), where thesettings, content, and transition/shifting implementation informationmight be stored in the form of a library in a database accessible by theplatform.

In some instances, the user experiences might be tailored to addresspsychological conditions of users, to implement exposure therapy forusers, or to perform cognitive behavior conditioning, and/or the like.In some cases, the system and platform might be used to address or evencure certain phobias that might be plaguing users (e.g., fear ofspiders, fear of open spaces, fear of heights, etc.), to help users copewith or thrive in particular settings (particularly if the user ishesitant or fearful in such settings; e.g., events or episodes thatcause post-traumatic stress disorder in a user, a user'sgeneral/specific fear of speaking in front of a crowd, a user's warinessof people or home/school environments due to past abuses or the like,etc.), and so forth.

Computer-Simulated Reality Platform

FIG. 6 is a schematic diagram illustrating a system 600 for implementinga computer-simulated reality platform that enables shifting from one ofvirtual reality, augmented reality, and mixed reality to another ofvirtual reality, augmented reality, and mixed reality, in accordancewith various embodiments. In FIG. 6, system 600 might comprise acomputing system 605, which might include, without limitation, at leastone of a server computer, a cloud computing system, or a computingsystem of the visual output device, and/or the like. In some cases,computing system 605 might correspond to computing system 110 of system100 of FIG. 1 or computing system 310 of system 300 of FIG. 3, or thelike. System 600 might further comprise an asset integration and controlsystem 610 (as described in detail above), which might becommunicatively coupled with the computing system 605 via a network 615(which might correspond to network 115 of system 100 of FIG. 1 ornetwork 315 of system 300 of FIG. 3, or the like). System 600 mightfurther comprise one or more visual output devices 620, which might eachinclude, but are not limited to, a VR headset, an AR headset, a set ofAR glasses, a tablet computer, a set of bionic contact lenses, a smartphone adapted as part of a VR headset, or a smart phone adapted as partof an AR system, and/or the like. The one or more visual output devices620 might be communicatively coupled with the computing system 605 viathe network 615.

System 600 might further comprise an asset production system 625 and aproduction management system 630 (both as described in detail above),each of which might be communicatively coupled with the computing system605 via the network 615. In some cases, system 600 might furthercomprise a database 635 that is accessible via network 615. Database635, in some cases, might correspond to database 120 of system 100 ofFIG. 1 or database 320 of system 300 of FIG. 3, or the like.

According to some aspects, the asset integration and control system 610,each of the one or more visual output devices 620, the asset productionsystem 625, and the production management system 630 mightcommunicatively couple with the computing system 605 (via the network615) via a first API 640 a, a second API 640 b, a third API 640 c, and afourth API 640 d, respectively (collectively, “APIs 640”). Theinteractions amongst the computing system 605, the asset integration andcontrol system 610, each of the one or more visual output devices 620,the asset production system 625, and the production management system630 (collectively, “the computer-simulated reality platform”) forimplementing shifting from one of VR, AR, or MxR to another of VR, AR,or MxR are described in detail with respect to FIGS. 7A-7F below.

We now turn to FIGS. 7A-7F (collectively, “FIG. 7”), which are flowdiagrams illustrating a method 700 for implementing a computer-simulatedreality platform that enables shifting from one of VR, AR, or MxR toanother of VR, AR, or MxR, in accordance with various embodiments. InFIG. 7, the process in method 700 continues from FIG. 7A to FIG. 7Bfollowing the circular marker, denoted “A,” continues from FIG. 7A toFIG. 7C following the circular marker, denoted “B,” continues from FIG.7A to FIG. 7D following the circular marker, denoted “C,” continues fromFIG. 7A to FIG. 7E following the circular marker, denoted “D,” returnsfrom FIG. 7E to FIG. 7A following the circular marker, denoted, “E,” andcontinues from each of FIGS. 7B-7D to FIG. 7F following the circularmarker, denoted “F.”

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method illustrated by FIG. 7can be implemented by or with (and, in some cases, are described belowwith respect to) the system 600 of FIG. 6 (or components thereof), suchmethod may also be implemented using any suitable hardware (or software)implementation. Similarly, while the system 600 of FIG. 6 (or componentsthereof) can operate according to the method illustrated by FIG. 7(e.g., by executing instructions embodied on a computer readablemedium), the system 600 of FIG. 6 can also operate according to othermodes of operation and/or perform other suitable procedures.

With reference to FIG. 7A, method 700 might comprise, at block 705,determining, with a computing system, whether one or more first mediacontent of a plurality of media content (which might include VR/AR/MxRassets, VR/AR/MxR content, and/or the like) is ready for distribution toa plurality of visual output devices. If so, the process continues toblock 710. If not, the process proceeds to block 765 in FIG. 7E.

At block 710, based on a determination that the one or more first mediacontent is ready for distribution, method 700 might comprise sending,with the computing system, one or more first messages to a productionmanagement system via a first API. The one or more first messages causesthe production management system to change a status of the one or morefirst media content to indicate that the one or more first media contentis ready for distribution to the plurality of visual output devices.Method 700, at block 715, might comprise sending, with the computingsystem, one or more second messages to an asset production system via asecond API. The one or more second messages causes the asset productionsystem to send the one or more first media content to the computingsystem via the second API.

Method 700 might further comprise receiving, with the computing system,the one or more first media content from the asset production system viathe second API (block 720) and receiving, with the computing system, theone or more captured images of the one or more physical objects from animage capture device (which might correspond to first camera 135 ofsystem 100 of FIG. 1, or the like) (block 725). At block 730, method 700might comprise determining, with the computing system, how at least oneof the one or more first media content of the plurality of media contentor the one or more captured images of the one or more physical objectsshould be presented to a user. The process might continue to one of theprocess at block 735 in FIG. 7B following marker “A” (for virtualreality presentation), the process at block 745 in FIG. 7C followingmarker “B” (for augmented reality presentation), or the process at block755 in FIG. 7D following marker “C” (for mixed reality presentation)

Turning to FIG. 7B, method 700 might comprise, based on a determinationto present the one or more first media content as a virtual realitypresentation, and based on a determination that the one or more firstmedia content is not already being presented as a virtual realitypresentation, controlling, with the computing system via a third API, anasset integration and control system to cause the asset integration andcontrol system to modify the first media content to generate one or moresecond media content, the one or more second media content comprisingvirtual reality content (block 735), and sending, with the computingsystem via a fourth API, the one or more second media content to a firstvisual output device of the plurality of visual output devices thatpresents the one or more second media content to the user (block 740).The process might continue to the process at optional block 775 in FIG.7F.

With reference to FIG. 7C, method 700 might comprise, based on adetermination to present the one or more first media content and the oneor more captured images of the one or more physical objects as anaugmented reality presentation, and based on a determination that theone or more first media content and the one or more captured images ofthe one or more physical objects are not already being presented as anaugmented reality presentation, controlling, with the computing systemvia the third API, the asset integration and control system to cause theasset integration and control system to generate one or more third mediacontent by modifying the one or more first media content as a virtualoverlay in relation to the one or more captured images of the one ormore physical objects, the one or more third media content comprisingaugmented reality content (block 745), and sending, with the computingsystem via the fourth API, the one or more third media content to thefirst visual output device that presents the one or more third mediacontent to the user (block 750). The process might continue to theprocess at optional block 775 in FIG. 7F.

In FIG. 7D, method 700 might comprise, based on a determination topresent the one or more first media content and the one or more capturedimages of the one or more physical objects as a mixed realitypresentation combining elements of virtual reality and augmentedreality, and based on a determination that the one or more first mediacontent and the one or more captured images of the one or more physicalobjects are not already being presented as a mixed reality presentation,controlling, with the computing system via the third API, the assetintegration and control system to cause the asset integration andcontrol system to generate one or more fourth media content based on theone or more first media content and the one or more captured images ofthe one or more physical objects, the one or more fourth media contentcomprising mixed reality content (block 755), and sending, with thecomputing system via the fourth API, the one or more fourth mediacontent to the first visual output device that presents the one or morefourth media content to the user (block 760). The process might continueto the process at optional block 775 in FIG. 7F.

Turning back to block 705 in FIG. 7A, based on a determination that theone or more first media content is not ready for distribution, method700 might proceed to block 765 in FIG. 7E following marker “D.” At block765, method 700 might comprise sending, with the computing system, oneor more third messages to the production management system via the firstAPI. The one or more third messages causes the production managementsystem to change a status of the one or more first media content toindicate that the one or more first media content is not ready fordistribution to the plurality of visual output devices. Method 700, atblock 770, might comprise sending, with the computing system, one ormore fourth messages to the asset production system via the second API,wherein the one or more fourth messages causes the asset productionsystem to send one or more fifth messages to one or more developers tocontinue work on the one or more first media content. The process mightreturn to the process at block 705 in FIG. 7A following marker “E.”

With reference to optional block 775 in FIG. 7F, which follows themarker “F” from each of blocks 740, 750, and 760 in FIGS. 7B, 7C, and7D, respectively, method 700 might further comprise tracking, with thecomputing system, a real-time status of each of the plurality of themedia content in each of at least one of the production managementsystem via the first API, the asset production system via the secondAPI, the asset integration and control system via the third API, andeach of the plurality of visual output devices via the fourth API.System 700 might further comprise distributing, with the computingsystem, at least one fifth media content of the plurality of the mediacontent whose status indicates that it is ready for integration from theasset production system to the asset integration and control system(optional block 780); distributing, with the computing system, at leastone sixth media content of the plurality of the media content whosestatus indicates that it is ready for distribution from the assetintegration and control system to the first visual output device(optional block 785); syncing, with the computing system, the at leastone sixth media content with one or more second visual output devices ofthe plurality of visual output devices, the one or more second visualoutput devices being separate from the first visual output device(optional block 790); and, in response to receiving updates to the atleast one sixth media content, sending, with the computing system, theupdates to the at least one sixth media content to each of the firstvisual output device and each of the one or more second visual outputdevices (optional block 795).

The system 600 is otherwise similar to, if not identical to, the systems100 and 300 of FIGS. 1 and 3, and the method 700 is otherwise similarto, if not identical to, the methods 200, 400, and 500 of FIGS. 2, 4,and 5, and descriptions of similar components or processes apply to thecomponents of system 600 and the processes of method 700.

Exemplary System and Hardware Implementation

FIG. 8 is a block diagram illustrating an exemplary computer or systemhardware architecture, in accordance with various embodiments. FIG. 8provides a schematic illustration of one embodiment of a computer system800 that can perform the methods provided by various other embodiments,as described herein, and/or can perform the functions of computer orhardware system (i.e., computing system 110 or 310, and/or the like), orof any other device (e.g., user device 105 or 305, etc.), as describedabove. It should be noted that FIG. 8 is meant only to provide ageneralized illustration of various components, of which one or more (ornone) of each may be utilized as appropriate. FIG. 8, therefore, broadlyillustrates how individual system elements may be implemented in arelatively separated or relatively more integrated manner.

The computer or hardware system 800 which might represent an embodimentof the computer or hardware system (i.e., computing system 110 or 310,and/or the like), or of any other device (e.g., user device 105 or 305,etc.), described above with respect to FIGS. 1 and 3—is shown comprisinghardware elements that can be electrically coupled via a bus 805 (or mayotherwise be in communication, as appropriate). The hardware elementsmay include one or more processors 810, including, without limitation,one or more general-purpose processors and/or one or morespecial-purpose processors (such as digital signal processing chips,graphics acceleration processors, and/or the like); one or more inputdevices 815, which can include, without limitation, a mouse, a keyboard,and/or the like; and one or more output devices 820, which can include,without limitation, a display device, a printer, and/or the like.

The computer or hardware system 800 may further include (and/or be incommunication with) one or more storage devices 825, which can comprise,without limitation, local and/or network accessible storage, and/or caninclude, without limitation, a disk drive, a drive array, an opticalstorage device, solid-state storage device such as a random accessmemory (“RAM”) and/or a read-only memory (“ROM”), which can beprogrammable, flash-updateable, and/or the like. Such storage devicesmay be configured to implement any appropriate data stores, including,without limitation, various file systems, database structures, and/orthe like.

The computer or hardware system 800 might also include a communicationssubsystem 830, which can include, without limitation, a modem, a networkcard (wireless or wired), an infra-red communication device, a wirelesscommunication device and/or chipset (such as a Bluetooth™ device, an802.11 device, a WiFi device, a-WiMax device, a WWAN device, cellularcommunication facilities, etc.), and/or the like. The communicationssubsystem 830 may permit data to be exchanged with a network (such asthe network described below, to name one example), with other computeror hardware systems, and/or with any other devices described herein. Inmany embodiments, the computer or hardware system 800 will furthercomprise a working memory 835, which can include a RAM or ROM device, asdescribed above.

The computer or hardware system 800 also may comprise software elements,shown as being currently located within the working memory 835,including an operating system 840, device drivers, executable libraries,and/or other code, such as one or more application programs 845, whichmay comprise computer programs provided by various embodiments(including, without limitation, hypervisors, VMs, and the like), and/ormay be designed to implement methods, and/or configure systems, providedby other embodiments, as described herein. Merely by way of example, oneor more procedures described with respect to the method(s) discussedabove might be implemented as code and/or instructions executable by acomputer (and/or a processor within a computer); in an aspect, then,such code and/or instructions can be used to configure and/or adapt ageneral purpose computer (or other device) to perform one or moreoperations in accordance with the described methods.

A set of these instructions and/or code might be encoded and/or storedon a non-transitory computer readable storage medium, such as thestorage device(s) 825 described above. In some cases, the storage mediummight be incorporated within a computer system, such as the system 800.In other embodiments, the storage medium might be separate from acomputer system (i.e., a removable medium, such as a compact disc,etc.), and/or provided in an installation package, such that the storagemedium can be used to program, configure, and/or adapt a general purposecomputer with the instructions/code stored thereon. These instructionsmight take the form of executable code, which is executable by thecomputer or hardware system 800 and/or might take the form of sourceand/or installable code, which, upon compilation and/or installation onthe computer or hardware system 800 (e.g., using any of a variety ofgenerally available compilers, installation programs,compression/decompression utilities, etc.) then takes the form ofexecutable code.

It will be apparent to those skilled in the art that substantialvariations may be made in accordance with specific requirements. Forexample, customized hardware (such as programmable logic controllers,field-programmable gate arrays, application-specific integratedcircuits, and/or the like) might also be used, and/or particularelements might be implemented in hardware, software (including portablesoftware, such as applets, etc.), or both. Further, connection to othercomputing devices such as network input/output devices may be employed.

As mentioned above, in one aspect, some embodiments may employ acomputer or hardware system (such as the computer or hardware system800) to perform methods in accordance with various embodiments of theinvention. According to a set of embodiments, some or all of theprocedures of such methods are performed by the computer or hardwaresystem 800 in response to processor 810 executing one or more sequencesof one or more instructions (which might be incorporated into theoperating system 840 and/or other code, such as an application program845) contained in the working memory 835. Such instructions may be readinto the working memory 835 from another computer readable medium, suchas one or more of the storage device(s) 825. Merely by way of example,execution of the sequences of instructions contained in the workingmemory 835 might cause the processor(s) 810 to perform one or moreprocedures of the methods described herein.

The terms “machine readable medium” and “computer readable medium,” asused herein, refer to any medium that participates in providing datathat causes a machine to operate in a specific fashion. In an embodimentimplemented using the computer or hardware system 800, various computerreadable media might be involved in providing instructions/code toprocessor(s) 810 for execution and/or might be used to store and/orcarry such instructions/code (e.g., as signals). In manyimplementations, a computer readable medium is a non-transitory,physical, and/or tangible storage medium. In some embodiments, acomputer readable medium may take many forms, including, but not limitedto, non-volatile media, volatile media, or the like. Non-volatile mediaincludes, for example, optical and/or magnetic disks, such as thestorage device(s) 825. Volatile media includes, without limitation,dynamic memory, such as the working memory 835. In some alternativeembodiments, a computer readable medium may take the form oftransmission media, which includes, without limitation, coaxial cables,copper wire and fiber optics, including the wires that comprise the bus805, as well as the various components of the communication subsystem830 (and/or the media by which the communications subsystem 830 providescommunication with other devices). In an alternative set of embodiments,transmission media can also take the form of waves (including, withoutlimitation, radio, acoustic, and/or light waves, such as those generatedduring radio-wave and infra-red data communications).

Common forms of physical and/or tangible computer readable mediainclude, for example, a floppy disk, a flexible disk, a hard disk,magnetic tape, or any other magnetic medium, a CD-ROM, any other opticalmedium, punch cards, paper tape, any other physical medium with patternsof holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chipor cartridge, a carrier wave as described hereinafter, or any othermedium from which a computer can read instructions and/or code.

Various forms of computer readable media may be involved in carrying oneor more sequences of one or more instructions to the processor(s) 810for execution. Merely by way of example, the instructions may initiallybe carried on a magnetic disk and/or optical disc of a remote computer.A remote computer might load the instructions into its dynamic memoryand send the instructions as signals over a transmission medium to bereceived and/or executed by the computer or hardware system 800. Thesesignals, which might be in the form of electromagnetic signals, acousticsignals, optical signals, and/or the like, are all examples of carrierwaves on which instructions can be encoded, in accordance with variousembodiments of the invention.

The communications subsystem 830 (and/or components thereof) generallywill receive the signals, and the bus 805 then might carry the signals(and/or the data, instructions, etc. carried by the signals) to theworking memory 835, from which the processor(s) 805 retrieves andexecutes the instructions. The instructions received by the workingmemory 835 may optionally be stored on a storage device 825 eitherbefore or after execution by the processor(s) 810.

As noted above, a set of embodiments comprises methods and systems forimplementing computer-simulated reality interactions, and, in particularembodiments, to methods, systems, and computer software for implementingcomputer-simulated reality interactions between users and physicalpublications. FIG. 9 illustrates a schematic diagram of a system 900that can be used in accordance with one set of embodiments. The system900 can include one or more user computer or user device 905. A usercomputer or user device 905 can be a general purpose personal computer(including, merely by way of example, desktop computers, tabletcomputers, laptop computers, handheld computers, and the like, runningany appropriate operating system, several of which are available fromvendors such as Apple, Microsoft Corp., and the like), cloud computingdevices, a server(s), and/or a workstation computer(s) running any of avariety of commercially-available UNIX“′” or UNIX-like operatingsystems. A user computer or user device 905 can also have any of avariety of applications, including one or more applications configuredto perform methods provided by various embodiments (as described above,for example), as well as one or more office applications, databaseclient and/or server applications, and/or web browser applications.Alternatively, a user computer or user device 905 can be any otherelectronic device, such as a thin-client computer, Internet-enabledmobile telephone, and/or personal digital assistant, capable ofcommunicating via a network (e.g., the network(s) 910 described below)and/or of displaying and navigating web pages or other types ofelectronic documents. Although the exemplary system 900 is shown withfour user computers or user devices 905, any number of user computers oruser devices can be supported.

Certain embodiments operate in a networked environment, which caninclude a network(s) 910. The network(s) 910 can be any type of networkfamiliar to those skilled in the art that can support datacommunications using any of a variety of commercially-available (and/orfree or proprietary) protocols, including, without limitation, TCP/IP,SNATM, AppleTalk™, and the like. Merely by way of example, thenetwork(s) 910 (similar to network 115 of FIG. 1 or network 315 of FIG.3, or the like) can each include a LAN, including, without limitation, afiber network, an Ethernet network, a Token-Ring™ network, and/or thelike; a WAN; a WWAN; a virtual network, such as a VPN; the Internet; anintranet; an extranet; a PSTN; an infra-red network; a wireless network,including, without limitation, a network operating under any of the IEEE802.11 suite of protocols, the Bluetooth™ protocol known in the art,and/or any other wireless protocol; and/or any combination of theseand/or other networks. In a particular embodiment, the network mightinclude an access network of the service provider (e.g., an ISP). Inanother embodiment, the network might include a core network of theservice provider, and/or the Internet.

Embodiments can also include one or more server computers 915. Each ofthe server computers 915 may be configured with an operating system,including, without limitation, any of those discussed above, as well asany commercially (or freely) available server operating systems. Each ofthe servers 915 may also be running one or more applications, which canbe configured to provide services to one or more clients 905 and/orother servers 915.

Merely by way of example, one of the servers 915 might be a data server,a web server, a cloud computing device(s), or the like, as describedabove. The data server might include (or be in communication with) a webserver, which can be used, merely by way of example, to process requestsfor web pages or other electronic documents from user computers 905. Theweb server can also run a variety of server applications, including HTTPservers, FTP servers, CGI servers, database servers, Java servers, andthe like. In some embodiments of the invention, the web server may beconfigured to serve web pages that can be operated within a web browseron one or more of the user computers 905 to perform methods of theinvention.

The server computers 915, in some embodiments, might include one or moreapplication servers, which can be configured with one or moreapplications accessible by a client running on one or more of the clientcomputers 905 and/or other servers 915. Merely by way of example, theserver(s) 915 can be one or more general purpose computers capable ofexecuting programs or scripts in response to the user computers 905and/or other servers 915, including, without limitation, webapplications (which might, in some cases, be configured to performmethods provided by various embodiments). Merely by way of example, aweb application can be implemented as one or more scripts or programswritten in any suitable programming language, such as Java™, C, C#™ orC++, and/or any scripting language, such as Perl, Python, or TCL, aswell as combinations of any programming and/or scripting languages. Theapplication server(s) can also include database servers, including,without limitation, those commercially available from Oracle™,Microsoft™, Sybase™ IBM™, and the like, which can process requests fromclients (including, depending on the configuration, dedicated databaseclients, API clients, web browsers, etc.) running on a user computer oruser device 905 and/or another server 915. In some embodiments, anapplication server can perform one or more of the processes forimplementing computer-simulated reality interactions, and, in particularembodiments, to methods, systems, and computer software for implementingcomputer-simulated reality interactions between users and physicalpublications, or the like, as described in detail above. Data providedby an application server may be formatted as one or more web pages(comprising HTML, JavaScript, etc., for example) and/or may be forwardedto a user computer 905 via a web server (as described above, forexample). Similarly, a web server might receive web page requests and/orinput data from a user computer 905 and/or forward the web page requestsand/or input data to an application server. In some cases, a web servermay be integrated with an application server.

In accordance with further embodiments, one or more servers 915 canfunction as a file server and/or can include one or more of the files(e.g., application code, data files, etc.) necessary to implementvarious disclosed methods, incorporated by an application running on auser computer 905 and/or another server 915. Alternatively, as thoseskilled in the art will appreciate, a file server can include allnecessary files, allowing such an application to be invoked remotely bya user computer or user device 905 and/or server 915.

It should be noted that the functions described with respect to variousservers herein (e.g., application server, database server, web server,file server, etc.) can be performed by a single server and/or aplurality of specialized servers, depending on implementation-specificneeds and parameters.

In certain embodiments, the system can include one or more databases 920a and 920 b (collectively, “databases 920”). The location of each of thedatabases 920 is discretionary: merely by way of example, a database 920a might reside on a storage medium local to (and/or resident in) aserver 915 a (and/or a user computer or user device 905). Alternatively,a database 920 b can be remote from any or all of the computers 905,915, so long as it can be in communication (e.g., via the network 910)with one or more of these. In a particular set of embodiments, adatabase 920 can reside in a storage-area network (“SAN”) familiar tothose skilled in the art. (Likewise, any necessary files for performingthe functions attributed to the computers 905, 915 can be stored locallyon the respective computer and/or remotely, as appropriate.) In one setof embodiments, the database 920 can be a relational database, such asan Oracle database, that is adapted to store, update, and retrieve datain response to SQL-formatted commands. The database might be controlledand/or maintained by a database server, as described above, for example.

According to some embodiments, system 900 might further comprise aprinted publication 925 (which might correspond to printed publication125 or 325 of FIGS. 1 and 3, respectively). In some embodiments, theprinted publication 925 might include, without limitation, one of afiction novel, a non-fiction novel, a children's book, a newspaper, amagazine, a comic book, a comic strip, a graphic novel, an encyclopedia,a dictionary, a map, a calendar, a greeting card, a bookmark, a venue orevent ticket stub, or a poster, and/or the like. In some cases, theprinted publication 925 might comprise one or more triggers 930 on atleast a portion of the printed publication 925. The one or moretriggers, in some instances, might each include, but is not limited to,one of a QR code provided on the at least a portion of the publication,a barcode provided on the at least a portion of the publication, agraphical code provided on a sticker affixed to the at least a portionof the publication, a graphic printed on the at least a portion of thepublication, a keyword or key code printed on the at least a portion ofthe publication, a cover image of the printed publication, a titleprinted on a cover of the printed publication, author name printed onthe cover of the printed publication, a stuffed character associatedwith the printed publication that is positioned in proximity to theprinted publication, a 3D-printed character associated with the printedpublication that is positioned in proximity to the printed publication,or a 3D-printed object associated with the printed publication that ispositioned in proximity to the printed publication, and/or the like.Alternatively, the one or more triggers might each include, withoutlimitation, one of a RFID tag, a BLE tag, a NFC tag, or a RTLS tag,and/or the like. The user device 905 might comprise one or more triggerdetectors 935, which might each include, without limitation, one of acamera, a wireless transceiver, and/or the like. The wirelesstransceiver might comprise one of an RFID scanner, a BLE interface, aNFC interface, a RTLS interface, and/or the like. System 900 mightfurther comprise a platform or computer-simulated reality platform 940,which enables shifting from one of VR, AR, or MxR to another of VR, AR,or MxR, and is described in detail with respect to FIGS. 6 and 7.

In operation, in response to at least one trigger detector 935 of theone or more trigger detector 935 detecting at least one trigger 930 ofthe one or more triggers 930, the user device 905 or the server 915might identify one or more media content that are associated with the atleast one trigger 930. In some embodiments, the one or more mediacontent comprises at least one of a 3D still graphic, a 3D animatedgraphic, or a 3D interactive animated graphic. In some instances, eachof the 3D still graphic, the 3D animated graphic, or the 3D interactiveanimated graphic comprises at least one of a graphic representation ofone or more characters associated with the printed publication, anon-fictional geographical location associated with the printedpublication, a fictional geographical location associated with theprinted publication, a room associated with the printed publication, abuilding associated with the printed publication, a scene described inthe printed publication, a scene inspired by the scene described in theprinted publication, one or more items described in the printedpublication, or one or more items inspired by the one or more itemsdescribed in the printed publication, and/or the like.

The user device 905 might determine a field of view of the user (e.g.,by using a camera such as camera 140 or 340 of FIGS. 1 and 3,respectively). The field of view might include at least a portion of theprinted publication. The user device 905 or the server 930 might, ifnecessary, modify an orientation of the one or more media content toalign with the field of view of the user, and might display, on adisplay device of the user device 905, the one or more media contentwhose orientation has been modified to align with the field of view ofthe user, in some cases, such that the one or more media content ispresented as a virtual overlay in relation to the at least a portion ofthe printed publication (e.g., presented as a virtual overlay above acover of the printed publication, presented as a virtual overlay on acover of the printed publication, presented as a virtual overlay besidethe printed publication, presented as a virtual overlay above a page ofthe printed publication, presented as a virtual overlay on a page of theprinted publication, presented as a virtual overlay in front of asurface of the printed publication, presented as a virtual overlay infront of a binding of the printed publication, presented as a virtualoverlay above a 2D graphic printed on a portion of the printedpublication, or presented as a virtual overlay on a 2D graphic printedon a portion of the printed publication, and the like).

Digital Processing Device

In some embodiments, the platforms, systems, media, and methodsdescribed herein include a digital processing device, or use of thesame. In further embodiments, the digital processing device includes oneor more hardware central processing units (CPUs) or general purposegraphics processing units (GPGPUs) that carry out the device'sfunctions. In still further embodiments, the digital processing devicefurther comprises an operating system configured to perform executableinstructions. In some embodiments, the digital processing device isoptionally connected a computer network. In further embodiments, thedigital processing device is optionally connected to the Internet suchthat it accesses the World Wide Web. In still further embodiments, thedigital processing device is optionally connected to a cloud computinginfrastructure. In other embodiments, the digital processing device isoptionally connected to an intranet. In other embodiments, the digitalprocessing device is optionally connected to a data storage device.

In accordance with the description herein, suitable digital processingdevices include, by way of non-limiting examples, server computers,desktop computers, laptop computers, notebook computers, sub-notebookcomputers, netbook computers, netpad computers, set-top computers, mediastreaming devices, handheld computers, Internet appliances, mobilesmartphones, tablet computers, personal digital assistants, video gameconsoles, and vehicles. Those of skill in the art will recognize thatmany smartphones are suitable for use in the system described herein.Those of skill in the art will also recognize that select televisions,video players, and digital music players with optional computer networkconnectivity are suitable for use in the system described herein.Suitable tablet computers include those with booklet, slate, andconvertible configurations, known to those of skill in the art.

In some embodiments, the digital processing device includes an operatingsystem configured to perform executable instructions. The operatingsystem is, for example, software, including programs and data, whichmanages the device's hardware and provides services for execution ofapplications. Those of skill in the art will recognize that suitableserver operating systems include, by way of non-limiting examples,FreeBSD, OpenBSD, NetBSD®, Linux, Apple® Mac OS X Server®, Oracle®Solaris®, Windows Server®, and Novell® NetWare®. Those of skill in theart will recognize that suitable personal computer operating systemsinclude, by way of non-limiting examples, Microsoft® Windows®, Apple®Mac OS X®, UNIX®, and UNIX-like operating systems such as GNU/Linux®. Insome embodiments, the operating system is provided by cloud computing.Those of skill in the art will also recognize that suitable mobile smartphone operating systems include, by way of non-limiting examples, Nokia®Symbian® OS, Apple® i0S®, Research In Motion® BlackBerry OS®, Google®Android®, Microsoft® Windows Phone® OS, Microsoft® Windows Mobile® OS,Linux®, and Palm® WebOS®. Those of skill in the art will also recognizethat suitable media streaming device operating systems include, by wayof non-limiting examples, Apple TV®, Roku®, Boxee®, Google TV®, GoogleChromecast®, Amazon Fire®, and Samsung® HomeSync®. Those of skill in theart will also recognize that suitable video game console operatingsystems include, by way of non-limiting examples, Sony® PS3®, Sony®PS4®, Microsoft® Xbox 360®, Microsoft Xbox One®, Nintendo® Wii®,Nintendo® Wii U®, and Ouya®.

In some embodiments, the device includes a storage and/or memory device.The storage and/or memory device is one or more physical apparatusesused to store data or programs on a temporary or permanent basis. Insome embodiments, the device is volatile memory and requires power tomaintain stored information. In some embodiments, the device isnon-volatile memory and retains stored information when the digitalprocessing device is not powered. In further embodiments, thenon-volatile memory comprises flash memory. In some embodiments, thenonvolatile memory comprises dynamic random-access memory (DRAM). Insome embodiments, the non-volatile memory comprises ferroelectric randomaccess memory (FRAM). In some embodiments, the non-volatile memorycomprises phase-change random access memory (PRAM). In otherembodiments, the device is a storage device including, by way ofnon-limiting examples, CD-ROMs, DVDs, flash memory devices, magneticdisk drives, magnetic tapes drives, optical disk drives, and cloudcomputing based storage. In further embodiments, the storage and/ormemory device is a combination of devices such as those disclosedherein.

In some embodiments, the digital processing device includes a display tosend visual information to a user. In some embodiments, the display is aliquid crystal display (LCD). In further embodiments, the display is athin film transistor liquid crystal display (TFT-LCD). In someembodiments, the display is an organic light emitting diode (OLED)display. In various further embodiments, on OLED display is apassive-matrix OLED (PMOLED) or active-matrix OLED (ANIOLED) display. Insome embodiments, the display is a plasma display. In other embodiments,the display is a video projector. In yet other embodiments, the displayis a head-mounted display in communication with the digital processingdevice, such as a VR headset. In further embodiments, suitable VRheadsets include, by way of non-limiting examples, HTC Vive, OculusRift, Samsung Gear VR, Microsoft HoloLens, Razer OSVR, FOVE VR, Zeiss VROne, Avegant Glyph, Freefly VR headset, and the like. In still furtherembodiments, the display is a combination of devices such as thosedisclosed herein.

In some embodiments, the digital processing device includes an inputdevice to receive information from a user. In some embodiments, theinput device is a keyboard. In some embodiments, the input device is apointing device including, by way of non-limiting examples, a mouse,trackball, track pad, joystick, game controller, or stylus. In someembodiments, the input device is a touch screen or a multi-touch screen.In other embodiments, the input device is a microphone to capture voiceor other sound input. In other embodiments, the input device is a videocamera or other sensor to capture motion or visual input. In furtherembodiments, the input device is a Kinect, Leap Motion, or the like. Instill further embodiments, the input device is a combination of devicessuch as those disclosed herein.

Non-Transitory Computer Readable Storage Medium

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include one or more non-transitory computer readablestorage media encoded with a program including instructions executableby the operating system of an optionally networked digital processingdevice. In further embodiments, a computer readable storage medium is atangible component of a digital processing device. In still furtherembodiments, a computer readable storage medium is optionally removablefrom a digital processing device. In some embodiments, a computerreadable storage medium includes, by way of non-limiting examples,CD-ROMs, DVDs, flash memory devices, solid state memory, magnetic diskdrives, magnetic tape drives, optical disk drives, cloud computingsystems and services, and the like. In some cases, the program andinstructions are permanently, substantially permanently,semi-permanently, or non-transitorily encoded on the media.

Computer Program

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include at least one computer program, or use of thesame. A computer program includes a sequence of instructions, executablein the digital processing device's CPU, written to perform a specifiedtask. Computer readable instructions may be implemented as programmodules, such as functions, objects, Application Programming Interfaces(APIs), data structures, and the like, that perform particular tasks orimplement particular abstract data types. In light of the disclosureprovided herein, those of skill in the art will recognize that acomputer program may be written in various versions of variouslanguages.

The functionality of the computer readable instructions may be combinedor distributed as desired in various environments. In some embodiments,a computer program comprises one sequence of instructions. In someembodiments, a computer program comprises a plurality of sequences ofinstructions. In some embodiments, a computer program is provided fromone location. In other embodiments, a computer program is provided froma plurality of locations. In various embodiments, a computer programincludes one or more software modules. In various embodiments, acomputer program includes, in part or in whole, one or more webapplications, one or more mobile applications, one or more standaloneapplications, one or more web browser plug-ins, extensions, add-ins, oradd-ons, or combinations thereof.

Web Application

In some embodiments, a computer program includes a web application. Inlight of the disclosure provided herein, those of skill in the art willrecognize that a web application, in various embodiments, utilizes oneor more software frameworks and one or more database systems. In someembodiments, a web application is created upon a software framework suchas Microsoft® .NET or Ruby on Rails (RoR). In some embodiments, a webapplication utilizes one or more database systems including, by way ofnon-limiting examples, relational, non-relational, object oriented,associative, and XML database systems. In further embodiments, suitablerelational database systems include, by way of non-limiting examples,Microsoft® SQL Server, mySQL™, and Oracle®. Those of skill in the artwill also recognize that a web application, in various embodiments, iswritten in one or more versions of one or more languages. A webapplication may be written in one or more markup languages, presentationdefinition languages, client-side scripting languages, server-sidecoding languages, database query languages, or combinations thereof. Insome embodiments, a web application is written to some extent in amarkup language such as Hypertext Markup Language (HTML), ExtensibleHypertext Markup Language (XHTML), or eXtensible Markup Language (XML).In some embodiments, a web application is written to some extent in apresentation definition language such as Cascading Style Sheets (CSS).In some embodiments, a web application is written to some extent in aclient-side scripting language such as Asynchronous Javascript and XML(AJAX), Flash® Actionscript, Javascript, or Silverlight®. In someembodiments, a web application is written to some extent in aserver-side coding language such as Active Server Pages (ASP),ColdFusion®, Perl, Java™, JavaServer Pages (JSP), Hypertext Preprocessor(PHP), Python™, Ruby, Tcl, Smalltalk, WebDNA®, or Groovy. In someembodiments, a web application is written to some extent in a databasequery language such as Structured Query Language (SQL). In someembodiments, a web application integrates enterprise server productssuch as IBM° Lotus Domino®. In some embodiments, a web applicationincludes a media player element. In various further embodiments, a mediaplayer element utilizes one or more of many suitable multimediatechnologies including, by way of non-limiting examples, Adobe® Flash®,HTML 5, Apple® QuickTime®, Microsoft® Silverlight®, Java™, and Unity®.

Mobile Application

In some embodiments, a computer program includes a mobile applicationprovided to a mobile digital processing device. In some embodiments, themobile application is provided to a mobile digital processing device atthe time it is manufactured. In other embodiments, the mobileapplication is provided to a mobile digital processing device via thecomputer network described herein.

In view of the disclosure provided herein, a mobile application iscreated by techniques known to those of skill in the art using hardware,languages, and development environments known to the art. Those of skillin the art will recognize that mobile applications are written inseveral languages. Suitable programming languages include, by way ofnon-limiting examples, C, C++, C#, Objective-C, Java™, Javascript,Pascal, Object Pascal, Python™, Ruby, VB.NET, WML, and XFITML/HTML withor without CSS, or combinations thereof.

Suitable mobile application development environments are available fromseveral sources. Commercially available development environmentsinclude, by way of non-limiting examples, AirplaySDK, alcheMo,Appcelerator®, Celsius, Bedrock, Flash Lite, NET Compact Framework,Rhomobile, and WorkLight Mobile Platform. Other development environmentsare available without cost including, by way of non-limiting examples,Lazarus, MobiFlex, MoSync, and Phonegap. Also, mobile devicemanufacturers distribute software developer kits including, by way ofnon-limiting examples, iPhone and iPad (iOS) SDK, Android™ SDK,BlackBerry® SDK, BREW SDK, Palm® OS SDK, Symbian SDK, webOS SDK, andWindows® Mobile SDK.

Those of skill in the art will recognize that several commercial forumsare available for distribution of mobile applications including, by wayof non-limiting examples, Apple® App Store, Google® Play, ChromeWebStore, BlackBerry® App World, App Store for Palm devices, App Catalogfor webOS, Windows® Marketplace for Mobile, Ovi Store for Nokia®devices, Samsung® Apps, and Nintendo® DSi Shop.

Standalone Application

In some embodiments, a computer program includes a standaloneapplication, which is a program that is run as an independent computerprocess, not an add-on to an existing process, e.g., not a plug-in.Those of skill in the art will recognize that standalone applicationsare often compiled. A compiler is a computer program(s) that transformssource code written in a programming language into binary object codesuch as assembly language or machine code. Suitable compiled programminglanguages include, by way of non-limiting examples, C, C++, Objective-C,COBOL, Delphi, Eiffel, Java™, Lisp, Python™, Visual Basic, and VB.NET,or combinations thereof. Compilation is often performed, at least inpart, to create an executable program. In some embodiments, a computerprogram includes one or more executable complied applications.

Web Browser Plug-in

In some embodiments, the computer program includes a web browser plug-in(e.g., extension, etc.). In computing, a plug-in is one or more softwarecomponents that add specific functionality to a larger softwareapplication. Makers of software applications support plug-ins to enablethird-party developers to create abilities which extend an application,to support easily adding new features, and to reduce the size of anapplication. When supported, plug-ins enable customizing thefunctionality of a software application. For example, plug-ins arecommonly used in web browsers to play video, generate interactivity,scan for viruses, and display particular file types. Those of skill inthe art will be familiar with several web browser plug-ins including,Adobe® Flash® Player, Microsoft® Silverlight®, and Apple® QuickTime®.

In view of the disclosure provided herein, those of skill in the artwill recognize that several plug-in frameworks are available that enabledevelopment of plug-ins in various programming languages, including, byway of non-limiting examples, C++, Delphi, Java™, PUP, Python™, and VBNET, or combinations thereof.

Web browsers (also called Internet browsers) are software applications,designed for use with network-connected digital processing devices, forretrieving, presenting, and traversing information resources on theWorld Wide Web. Suitable web browsers include, by way of non-limitingexamples, Microsoft® Internet Explorer®, Mozilla® Firefox®, Google®Chrome, Apple® Safari®, Opera Software® Opera®, and KDE Konqueror. Insome embodiments, the web browser is a mobile web browser. Mobile webbrowsers (also called mircrobrowsers, mini-browsers, and wirelessbrowsers) are designed for use on mobile digital processing devicesincluding, by way of non-limiting examples, handheld computers, tabletcomputers, netbook computers, subnotebook computers, smartphones, musicplayers, personal digital assistants (PDAs), and handheld video gamesystems. Suitable mobile web browsers include, by way of non-limitingexamples, Google® Android® browser, RIM BlackBerry® Browser, Apple®Safari®, Palm® Blazer, Palm® WebOS® Browser, Mozilla® Firefox® formobile, Microsoft® Internet Explorer® Mobile, Amazon® Kindle® Basic Web,Nokia® Browser, Opera Software® Opera® Mobile, and Sony® PSP™ browser.

Software Modules

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include software, server, and/or database modules, oruse of the same. In view of the disclosure provided herein, softwaremodules are created by techniques known to those of skill in the artusing machines, software, and languages known to the art. The softwaremodules disclosed herein are implemented in a multitude of ways. Invarious embodiments, a software module comprises a file, a section ofcode, a programming object, a programming structure, or combinationsthereof. In further various embodiments, a software module comprises aplurality of files, a plurality of sections of code, a plurality ofprogramming objects, a plurality of programming structures, orcombinations thereof. In various embodiments, the one or more softwaremodules comprise, by way of non-limiting examples, a web application, amobile application, and a standalone application. In some embodiments,software modules are in one computer program or application. In otherembodiments, software modules are in more than one computer program orapplication. In some embodiments, software modules are hosted on onemachine. In other embodiments, software modules are hosted on more thanone machine. In further embodiments, software modules are hosted oncloud computing platforms. In some embodiments, software modules arehosted on one or more machines in one location. In other embodiments,software modules are hosted on one or more machines in more than onelocation.

Databases

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include one or more databases, or use of the same. Inview of the disclosure provided herein, those of skill in the art willrecognize that many databases are suitable for storage and retrieval ofuser, publication, trigger, AR content, and VR content information. Invarious embodiments, suitable databases include, by way of non-limitingexamples, relational databases, non-relational databases, objectoriented databases, object databases, entity-relationship modeldatabases, associative databases, and XML databases. Furthernon-limiting examples include SQL, PostgreSQL, MySQL, Oracle, DB2, andSybase. In some embodiments, a database is internet-based. In furtherembodiments, a database is web-based. In still further embodiments, adatabase is cloud computing-based. In other embodiments, a database isbased on one or more local computer storage devices.

EXAMPLES

The following illustrative examples are representative of embodiments ofthe software applications, systems, and methods described herein and arenot meant to be limiting in any way.

Example I—A Physical Book with Triggers Initiating AR and VR UserExperiences

In a particular embodiment, a user obtains a publication in the form ofan e-book for young readers entitled “Morning Nightmare.” As depicted inFIGS. 10-25, the publication presents an illustrated story of amemorable winter school bus ride. The story presented in the publicationincludes messages about overcoming fears and building self-confidenceand highlights coping skills and conflict resolution techniques. Thestory is presented in the first person.

Referring to FIG. 19, the publication includes a QR code, which whenscanned by a mobile computing device of the user, provides access todownload and install an application as described herein. The applicationis also available via app stores known to the field. The application,once installed, allows the user to view their surroundings through acamera of the mobile computing device, the output of which is displayedon the screen of the device.

Referring to FIG. 26, the user installs the application and activatesthe application icon to launch the application. The user then viewstheir surroundings, with the application running on the device, via theoutput of the camera on the display screen of the mobile computingdevice. As such a live video feed of the user's surroundings isavailable to the application. Continuing to refer to FIG. 26, when aprintout of the e-book (or the onscreen presentation of the e-bookitself) is viewed via the device running the application, the cover artof the publication is identified as a trigger. In this case, an image ofthe cover art was uploaded to a database and associated with mediacontent.

Referring to FIG. 27, the application uses the identified trigger toquery the database and identify media content associated with thetrigger. The media content identified in this case is a 3D, animated,interactive media element. The media content is presented to the user onthe screen of the device as AR content overlaid on the publication'scover. Continuing to refer to FIG. 26, the AR content includes acharacter from the publication's story engaging in an activity relatedto the plot and theme of the story. In this case, a child from the buswho is antagonizing the user by throwing snowballs.

In some cases, the user can interact with the media content via themobile computing device's input elements such as a touchscreen,multitouch screen, buttons, stylus, trackball, or the like. In aparticular embodiment, the user can throw snowballs back at thecharacter in the media content by touching the character in the ARoverlay.

Referring to FIG. 28, the application continually monitors the field ofview of a user to determine if the publication and any triggers thereinare in the field of view and to determine the users viewing angle withrespect to the publication and the identified trigger. Continuing torefer to FIG. 28, when the user's view rotates with respect to thepublication and the identified trigger, the orientation of the mediacontent is modified to align with the field of view of the user and thecharacter in the AR media content turns to face the user.

Referring to FIG. 31, completion of the first enhanced interactiveexperience triggers a second, which is a VR experience. The user isinstructed to place their mobile device into an VR headset to view thesecond enhanced interactive experience. Continuing to refer to FIG. 31,a trigger is identified and used to further identify a second mediacontent. In this case, a VR media content depicting a 3D interactivescene associated with the story and presented from the first personperspective of the story.

FIGS. 32-36 demonstrate an alternative second enhanced interactiveexperience, again, VR media content depicting a scene related to thestory and presented from the first person perspective of the maincharacter of the story. In this case, a snowy scene illustratingmultiple characters from the story engaged in activities related to thestory. When the user's view rotates in space, the orientation of themedia content is modified to provide a 3D and 360 degree VR scene.

Referring to FIG. 34 in particular, the VR media content is interactive.By using a focal point marked at the center of the user's field of viewin the VR environment, the user is able to interact with the environmentand the characters using gaze placement and duration. In this case,again picking a snowball fight with a character from the story.

While certain features and aspects have been described with respect toexemplary embodiments, one skilled in the art will recognize thatnumerous modifications are possible. For example, the methods andprocesses described herein may be implemented using hardware components,software components, and/or any combination thereof. Further, whilevarious methods and processes described herein may be described withrespect to particular structural and/or functional components for easeof description, methods provided by various embodiments are not limitedto any particular structural and/or functional architecture but insteadcan be implemented on any suitable hardware, firmware and/or softwareconfiguration. Similarly, while certain functionality is ascribed tocertain system components, unless the context dictates otherwise, thisfunctionality can be distributed among various other system componentsin accordance with the several embodiments.

Moreover, while the procedures of the methods and processes describedherein are described in a particular order for ease of description,unless the context dictates otherwise, various procedures may bereordered, added, and/or omitted in accordance with various embodiments.Moreover, the procedures described with respect to one method or processmay be incorporated within other described methods or processes;likewise, system components described according to a particularstructural architecture and/or with respect to one system may beorganized in alternative structural architectures and/or incorporatedwithin other described systems. Hence, while various embodiments aredescribed with—or without-certain features for ease of description andto illustrate exemplary aspects of those embodiments, the variouscomponents and/or features described herein with respect to a particularembodiment can be substituted, added and/or subtracted from among otherdescribed embodiments, unless the context dictates otherwise.Consequently, although several exemplary embodiments are describedabove, it will be appreciated that the invention is intended to coverall modifications and equivalents within the scope of the followingclaims.

1. A computer-implemented method of enhancing user engagement with apublication comprising: receiving one or more live camera feedscomprising image data of a publication that includes a plurality oftriggers; identifying a first trigger in the live camera feed data thatprovides one or more augmented reality display media related to thepublication; presenting, via a visual output device, the one or moreaugmented reality display media; identifying a digital trigger in theone or more augmented reality display media, the digital triggerincluding a user interaction with a portion of the one or more augmentedreality display media; identifying one or more virtual realityenvironments related to the publication and associated with the digitaltrigger; terminating presentation of the one or more augmented realitydisplay media and presenting, via the visual output device, the one ormore virtual reality environments; identifying a second trigger in theone or more virtual reality environments, the second trigger including auser interaction with a portion of the one or more virtual realityenvironments; identifying a piece of content associated with the secondtrigger and related to the publication; and distributing the piece ofcontent to an application using an application programming interface. 2.The method of claim 1, further comprising identifying a pre-determinedcombination of triggers that includes the first trigger, the digitaltrigger, and the second trigger; identifying a piece of contentassociated with the pre-determined combination of triggers and relatedto the publication; and adapting the piece of content into an immersivescene included in the one or more virtual reality environments.
 3. Themethod of claim 1, wherein the publication is at least one of a poster,map, menu, billboard, program, or event ticket stub.
 4. The method ofclaim 1, wherein the augmented reality display media includes a threedimensional (3D) interactive animated graphic representation of one ormore characters associated with the publication.
 5. The method of claim1, wherein the augmented reality display media includes a 3D interactiveanimated graphic representation of at least one of a non-fictionalgeographical location associated with the publication, a fictionalgeographical location associated with the publication, a room associatedwith the publication, a building associated with the publication, ascene described in the publication, or a scene inspired by the scenedescribed in the publication.
 6. The method of claim 1, wherein thevisual output device includes at least one of a virtual reality headset,an augmented reality headset, a set of augmented reality glasses, atablet computer, a set of bionic contact lenses, a smart phone adaptedas part of a virtual reality headset, or a smart phone adapted as partof an augmented reality system.
 7. The method of claim 1, wherein thepublication is included in a set of two of more publications arranged ina particular order, each publication included in the set of two or morepublications including a plurality of triggers.
 8. The method of claim7, further comprising: determining an arrangement of the set of two ormore publications; and identifying a piece of content associated withthe arrangement of the set of two or more publications.
 9. The method ofclaim 7, further comprising: identifying, for each of the two or morepublications, a wireless communication between the visual output deviceand at least one of the plurality of triggers; generating a virtualrepresentation of a location associated with the set of two or morepublications based on identifying the wireless communication between thevisual output device and at least one of the plurality of triggersincluded in each of the two or more publications; determining a positionof the visual output device relative to an arrangement of the set of twoor more publications; and changing the virtual representation of thelocation as the position of the visual output device changes relative tothe arrangement of the set of two or more publications.
 10. Acomputer-implemented method of enhancing user engagement with apublication comprising: receiving one or more live camera feedscomprising image data of a publication that includes a plurality oftriggers; identifying at least one trigger in the live camera feed datathat provides one or more augmented reality display media related to thepublication; presenting, via a visual output device, the one or moreaugmented reality display media; identifying a digital trigger in theone or more augmented reality display media, the digital triggerincluding a user interaction with a portion of the one or more augmentedreality display media; identifying one or more virtual realityenvironments related to at least one of an object or a location includedin the publication and associated with the digital trigger, the one ormore virtual reality environments including one or more simulatedreality segments embedded in the one or more virtual realityenvironments, wherein the one or more simulated reality segments includea virtual simulation of at least one of the object or the location; andterminating presentation of the one or more augmented reality displaymedia and presenting via the visual output device the one or morevirtual reality environments.
 11. The method of claim 10, furthercomprising: identifying a second trigger in the one or more virtualreality environments, the second trigger including a user interactionwith a portion of the one or more virtual reality environments;identifying a piece of content associated with the second trigger andrelated to the publication; and distributing the piece of content to anapplication using an application programming interface.
 12. The methodof claim 10, wherein the one or more simulated reality segments includea physical reality representation, an augmented reality representation,and a virtual reality representation of at least one of the object orthe location.
 13. The method of claim 12, further comprising dynamicallyshifting between the augmented reality representation and the virtualreality representation seamlessly so that a user is unaware of atransition between the augmented reality representation and the virtualreality representation.
 14. A computer-implemented method of enhancinguser engagement with a publication comprising: receiving one or morelive camera feeds comprising image data of a publication that includes aplurality of triggers; identifying at least one trigger in the livecamera feed data and determining, via a visual output device, that auser is within a communications range of at least one non-visual triggerassociated with the publication; identifying one or more augmentedreality display media related to the publication and associated with apre-determined combination of triggers that includes the at least onetrigger and the at least one non-visual trigger; presenting, via thevisual output device, the one or more augmented reality display media;identifying a digital trigger in the one or more augmented realitydisplay media, the digital trigger including a user interaction with aportion of the one or more augmented reality display media; identifyingone or more virtual reality environments related to the publication andassociated with the digital trigger; and terminating presentation of theone or more augmented reality display media and presenting via thevisual output device, the one or more virtual reality environments. 15.The method of claim 14, wherein the publication is at least one of aposter, map, menu, billboard, program, or event ticket stub.
 16. Themethod of claim 14, wherein the augmented reality display media includesa three dimensional (3D) interactive animated graphic representation ofone or more characters associated with the publication.
 17. The methodof claim 14, wherein the augmented reality display media includes a 3Dinteractive animated graphic representation of at least one of anon-fictional geographical location associated with the publication, afictional geographical location associated with the publication, a roomassociated with the publication, a building associated with thepublication, a scene described in the publication, or a scene inspiredby the scene described in the publication.
 18. The method of claim 14,wherein the at least one non-visual trigger includes at least one of aradio frequency identification (RFID) tag, a Bluetooth low energy (BLE)tag, a near field communication (NFC) tag, a real-time locating systems(RTLS) tag, or location based service (LBS) active on a smartphone usedto identify or activate augmented reality, virtual reality, or mixedreality media content.
 19. A system comprising: a publication comprisinga plurality of triggers; and a visual output device including a camera,a processor and memory connected to the camera, and a plurality of linesof instructions so that the processor is configured to: receive one ormore live camera feeds from the camera, the one or more live camerafeeds comprising image data of the publication; identify at least onetrigger in the live camera feed data and determine that a user is withina communications range of at least one non-visual trigger associatedwith the publication; identify one or more augmented reality displaymedia related to the publication and associated with a pre-determinedcombination of triggers that includes the at least one trigger and theat least one non-visual trigger; present, via the visual output device,the one or more augmented reality display media; identify a digitaltrigger in the one or more augmented reality display media, the digitaltrigger including a user interaction with a portion of the one or moreaugmented reality display media; identify one or more virtual realityenvironments related to the publication and associated with the digitaltrigger; and terminate presentation of the one or more augmented realitydisplay media and present via the visual output device, the one or morevirtual reality environments.
 20. The system of claim 19, wherein thevisual output device includes at least one of a virtual reality headset,an augmented reality headset, a set of augmented reality glasses, atablet computer, a set of bionic contact lenses, a smart phone adaptedas part of a virtual reality headset, or a smart phone adapted as partof an augmented reality system.
 21. A computer-implemented method ofenhancing user engagement with a publication comprising: receiving oneor more live camera feeds comprising image data of a publication thatincludes a plurality of triggers; identifying a first trigger in thelive camera feed data that provides one or more augmented realitydisplay media related to the publication; presenting, via a visualoutput device, the one or more augmented reality display media;identifying a digital trigger in the one or more augmented realitydisplay media, the digital trigger including a user interaction with aportion of the one or more augmented reality display media; identifyingone or more virtual reality environments related to the publication andassociated with the digital trigger; distributing the one or morevirtual reality environments to an application using an applicationprogramming interface; and terminating presentation of the one or moreaugmented reality display media and presenting, via the visual outputdevice, the one or more virtual reality environments in the application.22. The method of claim 21, wherein the one or more augmented realitydisplay media and the one or more virtual reality environments arerelated to at least one of an object or a location included in thepublication; and wherein the one or more augmented reality display mediaand the one or more virtual reality environments are presented as partof a marketing campaign for the at least one of the object or thelocation.